[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:58:25 dist-test-slave-dhph krb5kdc[32050](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:58:25 dist-test-slave-dhph krb5kdc[32050](info): set up 2 sockets
Aug 06 02:58:25 dist-test-slave-dhph krb5kdc[32050](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260806 02:58:26.673619 32043 mini_kdc.cc:121] Time spent starting KDC: real 1.290s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/krb5kdc/test-user.keytab.
Aug 06 02:58:26 dist-test-slave-dhph krb5kdc[32050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:58:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:58:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.74.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab.
Entry for principal kudu/127.31.74.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab.
I20260806 02:58:26.945487 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:33625
--webserver_interface=127.31.74.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab
--principal=kudu/127.31.74.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:42295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.74.254:33625
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:58:27.361239 32063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:58:27.361649 32063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:58:27.361853 32063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:58:27.369801 32063 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:58:27.369990 32063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:58:27.370081 32063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:58:27.370178 32063 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:58:27.370251 32063 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:58:27.382262 32063 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:42295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.74.254:33625
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab
--principal=kudu/127.31.74.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:33625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.74.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:58:27.383881 32063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:58:27.385563 32063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:58:27.402868 32068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:58:27.402890 32069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:58:27.403221 32063 server_base.cc:1034] running on GCE node
W20260806 02:58:27.402987 32071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:58:27.404441 32063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:58:27.406078 32063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:58:27.407351 32063 hybrid_clock.cc:648] HybridClock initialized: now 1785985107407284 us; error 45 us; skew 500 ppm
I20260806 02:58:27.408094 32063 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:58:27 dist-test-slave-dhph krb5kdc[32050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985107, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:58:27.416733 32063 init.cc:377] Logged in from keytab as kudu/127.31.74.254@KRBTEST.COM (short username kudu)
I20260806 02:58:27.418891 32063 webserver.cc:460] Webserver started at http://127.31.74.254:36669/ using document root <none> and password file <none>
I20260806 02:58:27.420085 32063 fs_manager.cc:359] Metadata directory not provided
I20260806 02:58:27.420250 32063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:58:27.420614 32063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:58:27.426251 32063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/minicluster-data/master-0/data/instance:
uuid: "4fa7cff2b9404a89ba8bb89a46dcc5e6"
format_stamp: "Formatted at 2026-08-06 02:58:27 on dist-test-slave-dhph"
server_key: "112c15541152d7f622cb2aa520dc3091"
server_key_iv: "91121f895a27253a82a850e75f40045d"
server_key_version: "encryptionkey@0"
I20260806 02:58:27.427525 32063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/minicluster-data/master-0/wal/instance:
uuid: "4fa7cff2b9404a89ba8bb89a46dcc5e6"
format_stamp: "Formatted at 2026-08-06 02:58:27 on dist-test-slave-dhph"
server_key: "112c15541152d7f622cb2aa520dc3091"
server_key_iv: "91121f895a27253a82a850e75f40045d"
server_key_version: "encryptionkey@0"
I20260806 02:58:27.434391 32063 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.003s
I20260806 02:58:27.440227 32079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:58:27.442050 32063 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260806 02:58:27.442283 32063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/minicluster-data/master-0/wal
uuid: "4fa7cff2b9404a89ba8bb89a46dcc5e6"
format_stamp: "Formatted at 2026-08-06 02:58:27 on dist-test-slave-dhph"
server_key: "112c15541152d7f622cb2aa520dc3091"
server_key_iv: "91121f895a27253a82a850e75f40045d"
server_key_version: "encryptionkey@0"
I20260806 02:58:27.442528 32063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:58:27.462196 32063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:58:27.468026 32063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:58:27.468393 32063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:58:27.491545 32063 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.254:33625
I20260806 02:58:27.491556 32132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.254:33625 every 8 connection(s)
I20260806 02:58:27.493032 32063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
I20260806 02:58:27.498135 32133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:58:27.502692 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 32063
I20260806 02:58:27.503043 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/minicluster-data/master-0/wal/instance
I20260806 02:58:27.505268 32043 external_mini_cluster.cc:1428] Setting key 3b063f7e3b78fddc08e1008f0af61abb
I20260806 02:58:27.512007 32133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fa7cff2b9404a89ba8bb89a46dcc5e6: Bootstrap starting.
I20260806 02:58:27.515831 32133 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4fa7cff2b9404a89ba8bb89a46dcc5e6: Neither blocks nor log segments found. Creating new log.
I20260806 02:58:27.517058 32133 log.cc:824] T 00000000000000000000000000000000 P 4fa7cff2b9404a89ba8bb89a46dcc5e6: Log is configured to *not* fsync() on all Append() calls
I20260806 02:58:27.520428 32133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fa7cff2b9404a89ba8bb89a46dcc5e6: No bootstrap required, opened a new log
I20260806 02:58:27.532752 32133 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4fa7cff2b9404a89ba8bb89a46dcc5e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fa7cff2b9404a89ba8bb89a46dcc5e6" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 33625 } }
I20260806 02:58:27.533350 32133 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4fa7cff2b9404a89ba8bb89a46dcc5e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:58:27.533481 32133 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4fa7cff2b9404a89ba8bb89a46dcc5e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fa7cff2b9404a89ba8bb89a46dcc5e6, State: Initialized, Role: FOLLOWER
I20260806 02:58:27.534508 32133 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4fa7cff2b9404a89ba8bb89a46dcc5e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fa7cff2b9404a89ba8bb89a46dcc5e6" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 33625 } }
I20260806 02:58:27.534888 32133 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4fa7cff2b9404a89ba8bb89a46dcc5e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:58:27.535069 32133 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4fa7cff2b9404a89ba8bb89a46dcc5e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:58:27.535282 32133 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4fa7cff2b9404a89ba8bb89a46dcc5e6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:58:27.538444 32133 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4fa7cff2b9404a89ba8bb89a46dcc5e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fa7cff2b9404a89ba8bb89a46dcc5e6" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 33625 } }
I20260806 02:58:27.539173 32133 leader_election.cc:302] T 00000000000000000000000000000000 P 4fa7cff2b9404a89ba8bb89a46dcc5e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4fa7cff2b9404a89ba8bb89a46dcc5e6; no voters:
I20260806 02:58:27.540560 32133 leader_election.cc:288] T 00000000000000000000000000000000 P 4fa7cff2b9404a89ba8bb89a46dcc5e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:58:27.540743 32138 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4fa7cff2b9404a89ba8bb89a46dcc5e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:58:27.542153 32138 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4fa7cff2b9404a89ba8bb89a46dcc5e6 [term 1 LEADER]: Becoming Leader. State: Replica: 4fa7cff2b9404a89ba8bb89a46dcc5e6, State: Running, Role: LEADER
I20260806 02:58:27.543033 32138 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4fa7cff2b9404a89ba8bb89a46dcc5e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fa7cff2b9404a89ba8bb89a46dcc5e6" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 33625 } }
I20260806 02:58:27.543510 32133 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4fa7cff2b9404a89ba8bb89a46dcc5e6 [sys.catalog]: configured and running, proceeding with master startup.
Aug 06 02:58:27 dist-test-slave-dhph krb5kdc[32050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 02:58:27.548138 32139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fa7cff2b9404a89ba8bb89a46dcc5e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4fa7cff2b9404a89ba8bb89a46dcc5e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fa7cff2b9404a89ba8bb89a46dcc5e6" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 33625 } } }
I20260806 02:58:27.548573 32139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fa7cff2b9404a89ba8bb89a46dcc5e6 [sys.catalog]: This master's current role is: LEADER
I20260806 02:58:27.553260 32140 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fa7cff2b9404a89ba8bb89a46dcc5e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4fa7cff2b9404a89ba8bb89a46dcc5e6. Latest consensus state: current_term: 1 leader_uuid: "4fa7cff2b9404a89ba8bb89a46dcc5e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fa7cff2b9404a89ba8bb89a46dcc5e6" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 33625 } } }
I20260806 02:58:27.554766 32140 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fa7cff2b9404a89ba8bb89a46dcc5e6 [sys.catalog]: This master's current role is: LEADER
I20260806 02:58:27.562281 32146 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:58:27.566756 32135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:27.510842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55500 (local address 127.31.74.254:33625)
0806 02:58:27.512375 (+ 1533us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:27.512395 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:27.513862 (+ 1467us) server_negotiation.cc:407] Connection header received
0806 02:58:27.526753 (+ 12891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:27.526788 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:27.527458 (+ 670us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:27.527988 (+ 530us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:27.532020 (+ 4032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:27.534443 (+ 2423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:27.536834 (+ 2391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:27.538015 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:27.538281 (+ 266us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:27.548263 (+ 9982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:58:27.548284 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:58:27.548319 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:58:27.548411 (+ 92us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:58:27.557168 (+ 8757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:27.559149 (+ 1981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:27.559166 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:27.559194 (+ 28us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:27.560281 (+ 1087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:27.561157 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:27.561171 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:27.561176 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:27.562456 (+ 1280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:58:27.562918 (+ 462us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:27.564206 (+ 1288us) server_negotiation.cc:299] Negotiation successful
0806 02:58:27.564764 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1155,"thread_start_us":816,"threads_started":1}
I20260806 02:58:27.571846 32146 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:58:27.585461 32146 catalog_manager.cc:1347] Generated new cluster ID: a1dfc8451b914801a9c49e7b0ecafb38
I20260806 02:58:27.585602 32146 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:58:27.602638 32146 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:58:27.604264 32146 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:58:27.611594 32146 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4fa7cff2b9404a89ba8bb89a46dcc5e6: Generated new TSK 0
I20260806 02:58:27.612561 32146 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.74.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.193.keytab.
Entry for principal kudu/127.31.74.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.193.keytab.
I20260806 02:58:27.687927 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.193:0
--local_ip_for_outbound_sockets=127.31.74.193
--webserver_interface=127.31.74.193
--webserver_port=0
--tserver_master_addrs=127.31.74.254:33625
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.193.keytab
--principal=kudu/127.31.74.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:42295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:58:28.127359 32159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:58:28.127700 32159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:58:28.127816 32159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:58:28.136351 32159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:58:28.136484 32159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:58:28.136607 32159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.193
I20260806 02:58:28.149065 32159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:42295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.193.keytab
--principal=kudu/127.31.74.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.74.193
--webserver_port=0
--tserver_master_addrs=127.31.74.254:33625
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.193
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:58:28.150425 32159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:58:28.152184 32159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:58:28.168659 32167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:58:28.168679 32165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:58:28.168969 32164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:58:28.169844 32159 server_base.cc:1034] running on GCE node
I20260806 02:58:28.170506 32159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:58:28.171659 32159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:58:28.172884 32159 hybrid_clock.cc:648] HybridClock initialized: now 1785985108172808 us; error 61 us; skew 500 ppm
I20260806 02:58:28.173230 32159 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:58:28 dist-test-slave-dhph krb5kdc[32050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985108, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:58:28.177552 32159 init.cc:377] Logged in from keytab as kudu/127.31.74.193@KRBTEST.COM (short username kudu)
I20260806 02:58:28.179527 32159 webserver.cc:460] Webserver started at http://127.31.74.193:36015/ using document root <none> and password file <none>
I20260806 02:58:28.180352 32159 fs_manager.cc:359] Metadata directory not provided
I20260806 02:58:28.180522 32159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:58:28.180886 32159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:58:28.184490 32159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/minicluster-data/ts-0/data/instance:
uuid: "c89c64f1cf8c42ce90950a02c769b67b"
format_stamp: "Formatted at 2026-08-06 02:58:28 on dist-test-slave-dhph"
server_key: "1514a1a6333cb8914dc006419d0644eb"
server_key_iv: "9405a02a9664497f89e826f79d7b0496"
server_key_version: "encryptionkey@0"
I20260806 02:58:28.185648 32159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/minicluster-data/ts-0/wal/instance:
uuid: "c89c64f1cf8c42ce90950a02c769b67b"
format_stamp: "Formatted at 2026-08-06 02:58:28 on dist-test-slave-dhph"
server_key: "1514a1a6333cb8914dc006419d0644eb"
server_key_iv: "9405a02a9664497f89e826f79d7b0496"
server_key_version: "encryptionkey@0"
I20260806 02:58:28.192399 32159 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.008s
I20260806 02:58:28.197068 32174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:58:28.198824 32159 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260806 02:58:28.199023 32159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/minicluster-data/ts-0/wal
uuid: "c89c64f1cf8c42ce90950a02c769b67b"
format_stamp: "Formatted at 2026-08-06 02:58:28 on dist-test-slave-dhph"
server_key: "1514a1a6333cb8914dc006419d0644eb"
server_key_iv: "9405a02a9664497f89e826f79d7b0496"
server_key_version: "encryptionkey@0"
I20260806 02:58:28.199256 32159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:58:28.219100 32159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:58:28.224660 32159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:58:28.225021 32159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:58:28.226228 32159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:58:28.228053 32159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:58:28.228168 32159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:58:28.228350 32159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:58:28.228433 32159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:58:28.272027 32159 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.193:34761
I20260806 02:58:28.272068 32288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.193:34761 every 8 connection(s)
I20260806 02:58:28.273592 32159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
I20260806 02:58:28.278748 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 32159
I20260806 02:58:28.279064 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/minicluster-data/ts-0/wal/instance
I20260806 02:58:28.279942 32043 external_mini_cluster.cc:1428] Setting key 3f3e8b8c191692bb67ea2c6bb72c6ec1
Aug 06 02:58:28 dist-test-slave-dhph krb5kdc[32050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985108, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.193@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 02:58:28.300860 32291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:28.276775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.193:35091 (local address 127.31.74.254:33625)
0806 02:58:28.277325 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:28.277330 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:28.278694 (+ 1364us) server_negotiation.cc:407] Connection header received
0806 02:58:28.281283 (+ 2589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:28.281289 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:28.281390 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:28.281513 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:28.283544 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:28.284422 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:28.287093 (+ 2671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:28.287552 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:28.287662 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:28.292074 (+ 4412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:58:28.292083 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:58:28.292090 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:58:28.292140 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:58:28.296548 (+ 4408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:28.297596 (+ 1048us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:28.297605 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:28.297612 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:28.297799 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:28.298393 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:28.298401 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:28.298406 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:28.298939 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:58:28.299150 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:28.300041 (+ 891us) server_negotiation.cc:299] Negotiation successful
0806 02:58:28.300341 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":422,"thread_start_us":264,"threads_started":1}
I20260806 02:58:28.301581 32292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:28.277369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:33625 (local address 127.31.74.193:35091)
0806 02:58:28.278464 (+ 1095us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:28.278523 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0806 02:58:28.280995 (+ 2472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:58:28.281798 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:58:28.281807 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:58:28.282453 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:58:28.283332 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:28.283360 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:28.284599 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:28.284607 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:28.286919 (+ 2312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:28.286938 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:28.287797 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:28.287806 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:28.288139 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:28.289043 (+ 904us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:58:28.289069 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:58:28.291795 (+ 2726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:58:28.296773 (+ 4978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:58:28.296782 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:58:28.296797 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:58:28.297395 (+ 598us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:58:28.297960 (+ 565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:58:28.297964 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:58:28.297967 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:58:28.298234 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:58:28.299126 (+ 892us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:58:28.299134 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:58:28.299558 (+ 424us) client_negotiation.cc:769] Sending connection context
0806 02:58:28.299865 (+ 307us) client_negotiation.cc:240] Negotiation successful
0806 02:58:28.300319 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"thread_start_us":370,"threads_started":1}
WARNING: no policy specified for kudu/127.31.74.194@KRBTEST.COM; defaulting to no policy
I20260806 02:58:28.303572 32289 heartbeater.cc:347] Connected to a master server at 127.31.74.254:33625
I20260806 02:58:28.303936 32289 heartbeater.cc:464] Registering TS with master...
I20260806 02:58:28.305068 32289 heartbeater.cc:511] Master 127.31.74.254:33625 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.194@KRBTEST.COM" created.
I20260806 02:58:28.308133 32096 ts_manager.cc:194] Registered new tserver with Master: c89c64f1cf8c42ce90950a02c769b67b (127.31.74.193:34761)
I20260806 02:58:28.311072 32096 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.74.193@KRBTEST.COM'} at 127.31.74.193:35091
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.194.keytab.
Entry for principal kudu/127.31.74.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.194.keytab.
I20260806 02:58:28.340271 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.194:0
--local_ip_for_outbound_sockets=127.31.74.194
--webserver_interface=127.31.74.194
--webserver_port=0
--tserver_master_addrs=127.31.74.254:33625
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.194.keytab
--principal=kudu/127.31.74.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:42295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:58:28.739431 32295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:58:28.739822 32295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:58:28.740005 32295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:58:28.748616 32295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:58:28.748821 32295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:58:28.749050 32295 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.194
I20260806 02:58:28.760973 32295 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:42295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.194.keytab
--principal=kudu/127.31.74.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.74.194
--webserver_port=0
--tserver_master_addrs=127.31.74.254:33625
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.194
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:58:28.762567 32295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:58:28.764067 32295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:58:28.779117 32301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:58:28.779129 32300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:58:28.779153 32303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:58:28.780143 32295 server_base.cc:1034] running on GCE node
I20260806 02:58:28.780915 32295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:58:28.782011 32295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:58:28.783299 32295 hybrid_clock.cc:648] HybridClock initialized: now 1785985108783239 us; error 50 us; skew 500 ppm
I20260806 02:58:28.783684 32295 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:58:28 dist-test-slave-dhph krb5kdc[32050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985108, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:58:28.787871 32295 init.cc:377] Logged in from keytab as kudu/127.31.74.194@KRBTEST.COM (short username kudu)
I20260806 02:58:28.789701 32295 webserver.cc:460] Webserver started at http://127.31.74.194:45083/ using document root <none> and password file <none>
I20260806 02:58:28.790508 32295 fs_manager.cc:359] Metadata directory not provided
I20260806 02:58:28.790747 32295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:58:28.791074 32295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:58:28.794391 32295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/minicluster-data/ts-1/data/instance:
uuid: "e0ca6972d43743c99f75a4e13b84f931"
format_stamp: "Formatted at 2026-08-06 02:58:28 on dist-test-slave-dhph"
server_key: "4d64ed8f0c0df34514e5ef577d8a629d"
server_key_iv: "197bfd1dd2de0e1afde73381b534e9a3"
server_key_version: "encryptionkey@0"
I20260806 02:58:28.795501 32295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/minicluster-data/ts-1/wal/instance:
uuid: "e0ca6972d43743c99f75a4e13b84f931"
format_stamp: "Formatted at 2026-08-06 02:58:28 on dist-test-slave-dhph"
server_key: "4d64ed8f0c0df34514e5ef577d8a629d"
server_key_iv: "197bfd1dd2de0e1afde73381b534e9a3"
server_key_version: "encryptionkey@0"
I20260806 02:58:28.802282 32295 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.002s
I20260806 02:58:28.806526 32310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:58:28.808521 32295 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 02:58:28.808758 32295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/minicluster-data/ts-1/wal
uuid: "e0ca6972d43743c99f75a4e13b84f931"
format_stamp: "Formatted at 2026-08-06 02:58:28 on dist-test-slave-dhph"
server_key: "4d64ed8f0c0df34514e5ef577d8a629d"
server_key_iv: "197bfd1dd2de0e1afde73381b534e9a3"
server_key_version: "encryptionkey@0"
I20260806 02:58:28.809013 32295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:58:28.827344 32295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:58:28.833174 32295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:58:28.833544 32295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:58:28.835002 32295 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:58:28.836926 32295 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:58:28.837014 32295 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:58:28.837206 32295 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:58:28.837286 32295 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:58:28.880470 32295 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.194:46563
I20260806 02:58:28.880523 32423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.194:46563 every 8 connection(s)
I20260806 02:58:28.881834 32295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
I20260806 02:58:28.888635 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 32295
I20260806 02:58:28.888908 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/minicluster-data/ts-1/wal/instance
I20260806 02:58:28.889681 32043 external_mini_cluster.cc:1428] Setting key 674ec7a52627d96f3ecfc57d57a048b7
Aug 06 02:58:28 dist-test-slave-dhph krb5kdc[32050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985108, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.194@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 02:58:28.909508 32426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:28.884933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.194:57593 (local address 127.31.74.254:33625)
0806 02:58:28.885725 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:28.885731 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:28.887252 (+ 1521us) server_negotiation.cc:407] Connection header received
0806 02:58:28.889338 (+ 2086us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:28.889344 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:28.889446 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:28.889572 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:28.892329 (+ 2757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:28.893631 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:28.895651 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:28.896401 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:28.896560 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:28.901349 (+ 4789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:58:28.901355 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:58:28.901358 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:58:28.901397 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:58:28.904744 (+ 3347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:28.906114 (+ 1370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:28.906121 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:28.906124 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:28.906404 (+ 280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:28.907274 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:28.907280 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:28.907283 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:28.907641 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:58:28.907835 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:28.908967 (+ 1132us) server_negotiation.cc:299] Negotiation successful
0806 02:58:28.909188 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":659,"thread_start_us":352,"threads_started":1}
I20260806 02:58:28.911448 32427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:28.886075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:33625 (local address 127.31.74.194:57593)
0806 02:58:28.887078 (+ 1003us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:28.887121 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 02:58:28.889040 (+ 1919us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:58:28.890203 (+ 1163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:58:28.890219 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:58:28.891199 (+ 980us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:58:28.892076 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:28.892104 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:28.894044 (+ 1940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:28.894048 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:28.895472 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:28.895486 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:28.896673 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:28.896696 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:28.897134 (+ 438us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:28.897971 (+ 837us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:58:28.897997 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:58:28.901035 (+ 3038us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:58:28.904922 (+ 3887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:58:28.904935 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:58:28.904959 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:58:28.905706 (+ 747us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:58:28.906699 (+ 993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:58:28.906707 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:58:28.906715 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:58:28.907060 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:58:28.907804 (+ 744us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:58:28.907817 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:58:28.908419 (+ 602us) client_negotiation.cc:769] Sending connection context
0806 02:58:28.908820 (+ 401us) client_negotiation.cc:240] Negotiation successful
0806 02:58:28.909575 (+ 755us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":740,"thread_start_us":490,"threads_started":1}
I20260806 02:58:28.913794 32424 heartbeater.cc:347] Connected to a master server at 127.31.74.254:33625
WARNING: no policy specified for kudu/127.31.74.195@KRBTEST.COM; defaulting to no policy
I20260806 02:58:28.914409 32424 heartbeater.cc:464] Registering TS with master...
I20260806 02:58:28.915318 32424 heartbeater.cc:511] Master 127.31.74.254:33625 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.195@KRBTEST.COM" created.
I20260806 02:58:28.917964 32096 ts_manager.cc:194] Registered new tserver with Master: e0ca6972d43743c99f75a4e13b84f931 (127.31.74.194:46563)
I20260806 02:58:28.918955 32096 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.74.194@KRBTEST.COM'} at 127.31.74.194:57593
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.195.keytab.
Entry for principal kudu/127.31.74.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.195.keytab.
I20260806 02:58:28.947706 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.195:0
--local_ip_for_outbound_sockets=127.31.74.195
--webserver_interface=127.31.74.195
--webserver_port=0
--tserver_master_addrs=127.31.74.254:33625
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.195.keytab
--principal=kudu/127.31.74.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:42295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:58:29.315393 32289 heartbeater.cc:503] Master 127.31.74.254:33625 was elected leader, sending a full tablet report...
W20260806 02:58:29.338693 32430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:58:29.339037 32430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:58:29.339149 32430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:58:29.348129 32430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:58:29.348346 32430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:58:29.348663 32430 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.195
I20260806 02:58:29.362057 32430 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:42295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.195.keytab
--principal=kudu/127.31.74.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.74.195
--webserver_port=0
--tserver_master_addrs=127.31.74.254:33625
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.195
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:58:29.363790 32430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:58:29.365355 32430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:58:29.381592 32436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:58:29.381707 32435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:58:29.381681 32438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:58:29.382788 32430 server_base.cc:1034] running on GCE node
I20260806 02:58:29.383560 32430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:58:29.384927 32430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:58:29.386409 32430 hybrid_clock.cc:648] HybridClock initialized: now 1785985109386333 us; error 60 us; skew 500 ppm
I20260806 02:58:29.386864 32430 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:58:29 dist-test-slave-dhph krb5kdc[32050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985109, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:58:29.390492 32430 init.cc:377] Logged in from keytab as kudu/127.31.74.195@KRBTEST.COM (short username kudu)
I20260806 02:58:29.392443 32430 webserver.cc:460] Webserver started at http://127.31.74.195:41725/ using document root <none> and password file <none>
I20260806 02:58:29.393338 32430 fs_manager.cc:359] Metadata directory not provided
I20260806 02:58:29.393503 32430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:58:29.393874 32430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:58:29.397274 32430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/minicluster-data/ts-2/data/instance:
uuid: "aa8a312f7d694381be658c14fa6fd234"
format_stamp: "Formatted at 2026-08-06 02:58:29 on dist-test-slave-dhph"
server_key: "3081fcee30380c52678f486d2ea8d591"
server_key_iv: "c02b6c98045ddc2083b40a5e639690db"
server_key_version: "encryptionkey@0"
I20260806 02:58:29.398237 32430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/minicluster-data/ts-2/wal/instance:
uuid: "aa8a312f7d694381be658c14fa6fd234"
format_stamp: "Formatted at 2026-08-06 02:58:29 on dist-test-slave-dhph"
server_key: "3081fcee30380c52678f486d2ea8d591"
server_key_iv: "c02b6c98045ddc2083b40a5e639690db"
server_key_version: "encryptionkey@0"
I20260806 02:58:29.405097 32430 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260806 02:58:29.410274 32446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:58:29.411981 32430 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260806 02:58:29.412227 32430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/minicluster-data/ts-2/wal
uuid: "aa8a312f7d694381be658c14fa6fd234"
format_stamp: "Formatted at 2026-08-06 02:58:29 on dist-test-slave-dhph"
server_key: "3081fcee30380c52678f486d2ea8d591"
server_key_iv: "c02b6c98045ddc2083b40a5e639690db"
server_key_version: "encryptionkey@0"
I20260806 02:58:29.412478 32430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:58:29.433130 32430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:58:29.439376 32430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:58:29.439913 32430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:58:29.441308 32430 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:58:29.443324 32430 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:58:29.443501 32430 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:58:29.443691 32430 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:58:29.443791 32430 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:58:29.488094 32430 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.195:36619
I20260806 02:58:29.488155 32560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.195:36619 every 8 connection(s)
I20260806 02:58:29.489534 32430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
I20260806 02:58:29.495827 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 32430
I20260806 02:58:29.496048 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785985105255475-32043-0/minicluster-data/ts-2/wal/instance
I20260806 02:58:29.496513 32043 external_mini_cluster.cc:1428] Setting key 1aabd6c41a1226784da562470482ffbb
Aug 06 02:58:29 dist-test-slave-dhph krb5kdc[32050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985109, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.195@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 02:58:29.516009 32563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:29.492624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.195:38375 (local address 127.31.74.254:33625)
0806 02:58:29.493146 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:29.493152 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:29.494327 (+ 1175us) server_negotiation.cc:407] Connection header received
0806 02:58:29.497207 (+ 2880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:29.497214 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:29.497316 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:29.497450 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:29.499520 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:29.500383 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:29.502494 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:29.503077 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:29.503413 (+ 336us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:29.507547 (+ 4134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:58:29.507553 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:58:29.507558 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:58:29.507610 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:58:29.510534 (+ 2924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:29.511537 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:29.511541 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:29.511544 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:29.511898 (+ 354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:29.512825 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:29.512832 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:29.512836 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:29.513512 (+ 676us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:58:29.514248 (+ 736us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:29.515451 (+ 1203us) server_negotiation.cc:299] Negotiation successful
0806 02:58:29.515657 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":235,"threads_started":1}
I20260806 02:58:29.517043 32564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:29.493268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:33625 (local address 127.31.74.195:38375)
0806 02:58:29.494119 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:29.494174 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 02:58:29.496855 (+ 2681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:58:29.497799 (+ 944us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:58:29.497807 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:58:29.498527 (+ 720us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:58:29.499341 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:29.499361 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:29.500551 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:29.500555 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:29.502313 (+ 1758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:29.502328 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:29.503562 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:29.503570 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:29.503829 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:29.504608 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:58:29.504633 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:58:29.507236 (+ 2603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:58:29.510796 (+ 3560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:58:29.510805 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:58:29.510820 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:58:29.511345 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:58:29.512205 (+ 860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:58:29.512229 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:58:29.512233 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:58:29.512573 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:58:29.514185 (+ 1612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:58:29.514196 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:58:29.514959 (+ 763us) client_negotiation.cc:769] Sending connection context
0806 02:58:29.515311 (+ 352us) client_negotiation.cc:240] Negotiation successful
0806 02:58:29.515769 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":332,"threads_started":1}
I20260806 02:58:29.518735 32561 heartbeater.cc:347] Connected to a master server at 127.31.74.254:33625
I20260806 02:58:29.519032 32561 heartbeater.cc:464] Registering TS with master...
I20260806 02:58:29.519775 32561 heartbeater.cc:511] Master 127.31.74.254:33625 requested a full tablet report, sending...
I20260806 02:58:29.521394 32096 ts_manager.cc:194] Registered new tserver with Master: aa8a312f7d694381be658c14fa6fd234 (127.31.74.195:36619)
I20260806 02:58:29.522257 32096 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.74.195@KRBTEST.COM'} at 127.31.74.195:38375
I20260806 02:58:29.528499 32043 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 02:58:29 dist-test-slave-dhph krb5kdc[32050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985109, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 02:58:29 dist-test-slave-dhph krb5kdc[32050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985109, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.74.193@KRBTEST.COM
I20260806 02:58:29.592525 32568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:29.573455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54886 (local address 127.31.74.193:34761)
0806 02:58:29.574173 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:29.574184 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:29.574801 (+ 617us) server_negotiation.cc:407] Connection header received
0806 02:58:29.574960 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:29.574970 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:29.575329 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:29.575540 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:29.577421 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:29.578503 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:29.580113 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:29.580710 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:29.580823 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:29.585612 (+ 4789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:58:29.585627 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:58:29.585642 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:58:29.585684 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:58:29.589000 (+ 3316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:29.589956 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:29.589963 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:29.589965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:29.590127 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:29.590650 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:29.590659 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:29.590663 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:29.591201 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:58:29.591486 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:29.591961 (+ 475us) server_negotiation.cc:299] Negotiation successful
0806 02:58:29.592211 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":344,"threads_started":1}
W20260806 02:58:29.594331 32202 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54886
Aug 06 02:58:29 dist-test-slave-dhph krb5kdc[32050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 02:58:29 dist-test-slave-dhph krb5kdc[32050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.74.193@KRBTEST.COM
I20260806 02:58:29.654127 32568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:29.637313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54894 (local address 127.31.74.193:34761)
0806 02:58:29.637598 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:29.637606 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:29.637941 (+ 335us) server_negotiation.cc:407] Connection header received
0806 02:58:29.638189 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:29.638194 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:29.638272 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:29.638396 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:29.640064 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:29.640871 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:29.642329 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:29.642890 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:29.643041 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:29.647805 (+ 4764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:58:29.647810 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:58:29.647814 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:58:29.647857 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:58:29.650856 (+ 2999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:29.651569 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:29.651576 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:29.651579 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:29.651916 (+ 337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:29.652412 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:29.652417 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:29.652420 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:29.652705 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:58:29.653010 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:29.653387 (+ 377us) server_negotiation.cc:299] Negotiation successful
0806 02:58:29.653645 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260806 02:58:29.656414 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 32159
I20260806 02:58:29.673189 32568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:29.657835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54906 (local address 127.31.74.193:34761)
0806 02:58:29.658076 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:29.658082 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:29.658988 (+ 906us) server_negotiation.cc:407] Connection header received
0806 02:58:29.659284 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:29.659289 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:29.659365 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:29.659471 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:29.661036 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:29.661908 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:29.663626 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:29.664139 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:29.664287 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:29.667104 (+ 2817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:58:29.667108 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:58:29.667112 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:58:29.667149 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:58:29.670256 (+ 3107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:29.671258 (+ 1002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:29.671262 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:29.671265 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:29.671419 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:29.671884 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:29.671888 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:29.671890 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:29.672171 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:58:29.672361 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:29.672691 (+ 330us) server_negotiation.cc:299] Negotiation successful
0806 02:58:29.672910 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260806 02:58:29.673961 32284 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:58:29.762928 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 32159
I20260806 02:58:29.783815 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 32295
Aug 06 02:58:29 dist-test-slave-dhph krb5kdc[32050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.74.194@KRBTEST.COM
I20260806 02:58:29.802381 32577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:29.784909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36926 (local address 127.31.74.194:46563)
0806 02:58:29.785504 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:29.785511 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:29.785539 (+ 28us) server_negotiation.cc:407] Connection header received
0806 02:58:29.785602 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:29.785609 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:29.785834 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:29.785968 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:29.787351 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:29.788402 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:29.789807 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:29.790380 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:29.790494 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:29.795168 (+ 4674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:58:29.795183 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:58:29.795198 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:58:29.795276 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:58:29.799070 (+ 3794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:29.799917 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:29.799926 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:29.799929 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:29.800080 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:29.800568 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:29.800572 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:29.800574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:29.801029 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:58:29.801323 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:29.801687 (+ 364us) server_negotiation.cc:299] Negotiation successful
0806 02:58:29.801944 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":316,"threads_started":1}
I20260806 02:58:29.803814 32419 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:58:29.897115 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 32295
I20260806 02:58:29.916065 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 32430
Aug 06 02:58:29 dist-test-slave-dhph krb5kdc[32050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.74.195@KRBTEST.COM
I20260806 02:58:29.936630 32580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:29.917613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60168 (local address 127.31.74.195:36619)
0806 02:58:29.918166 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:29.918173 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:29.918212 (+ 39us) server_negotiation.cc:407] Connection header received
0806 02:58:29.918276 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:29.918283 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:29.918498 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:29.918721 (+ 223us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:29.920090 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:29.921275 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:29.923149 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:29.924104 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:29.924316 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:29.929703 (+ 5387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:58:29.929727 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:58:29.929745 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:58:29.929842 (+ 97us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:58:29.933144 (+ 3302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:29.934081 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:29.934089 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:29.934092 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:29.934281 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:29.934851 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:29.934855 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:29.934857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:29.935260 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:58:29.935521 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:29.935932 (+ 411us) server_negotiation.cc:299] Negotiation successful
0806 02:58:29.936177 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":267,"threads_started":1}
I20260806 02:58:29.938092 32556 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:58:30.038723 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 32430
I20260806 02:58:30.061334 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 32063
I20260806 02:58:30.063172 32128 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:58:30.176810 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 32063
2026-08-06T02:58:30Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4860 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:58:30 dist-test-slave-dhph krb5kdc[32585](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:58:30 dist-test-slave-dhph krb5kdc[32585](info): set up 2 sockets
Aug 06 02:58:30 dist-test-slave-dhph krb5kdc[32585](info): commencing operation
krb5kdc: starting...
W20260806 02:58:31.373391 32043 mini_kdc.cc:121] Time spent starting KDC: real 1.145s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/krb5kdc/test-user.keytab.
Aug 06 02:58:31 dist-test-slave-dhph krb5kdc[32585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:58:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:58:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.74.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab.
Entry for principal kudu/127.31.74.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab.
I20260806 02:58:31.661010 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:45541
--webserver_interface=127.31.74.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab
--principal=kudu/127.31.74.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:46179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.74.254:45541
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:58:32.053731 32598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:58:32.054082 32598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:58:32.054301 32598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:58:32.062322 32598 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:58:32.062548 32598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:58:32.062667 32598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:58:32.062860 32598 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:58:32.062955 32598 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:58:32.073518 32598 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:46179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.74.254:45541
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab
--principal=kudu/127.31.74.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:45541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.74.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:58:32.074991 32598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:58:32.076414 32598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:58:32.090092 32606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:58:32.090142 32603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:58:32.090888 32604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:58:32.091154 32598 server_base.cc:1034] running on GCE node
I20260806 02:58:32.092090 32598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:58:32.093787 32598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:58:32.095070 32598 hybrid_clock.cc:648] HybridClock initialized: now 1785985112094995 us; error 67 us; skew 500 ppm
I20260806 02:58:32.095506 32598 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:58:32 dist-test-slave-dhph krb5kdc[32585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985112, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:58:32.099185 32598 init.cc:377] Logged in from keytab as kudu/127.31.74.254@KRBTEST.COM (short username kudu)
I20260806 02:58:32.101011 32598 webserver.cc:460] Webserver started at http://127.31.74.254:32925/ using document root <none> and password file <none>
I20260806 02:58:32.101799 32598 fs_manager.cc:359] Metadata directory not provided
I20260806 02:58:32.101959 32598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:58:32.102322 32598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:58:32.106207 32598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/minicluster-data/master-0/data/instance:
uuid: "42f81ed814ed4066a5071cd2c685c8e2"
format_stamp: "Formatted at 2026-08-06 02:58:32 on dist-test-slave-dhph"
server_key: "fac596657fd7dbd61606c281e0b7e2ae"
server_key_iv: "21fe137ea116d2128b14d73a39516e13"
server_key_version: "encryptionkey@0"
I20260806 02:58:32.107324 32598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/minicluster-data/master-0/wal/instance:
uuid: "42f81ed814ed4066a5071cd2c685c8e2"
format_stamp: "Formatted at 2026-08-06 02:58:32 on dist-test-slave-dhph"
server_key: "fac596657fd7dbd61606c281e0b7e2ae"
server_key_iv: "21fe137ea116d2128b14d73a39516e13"
server_key_version: "encryptionkey@0"
I20260806 02:58:32.114424 32598 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260806 02:58:32.119266 32613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:58:32.121179 32598 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.002s
I20260806 02:58:32.121415 32598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/minicluster-data/master-0/wal
uuid: "42f81ed814ed4066a5071cd2c685c8e2"
format_stamp: "Formatted at 2026-08-06 02:58:32 on dist-test-slave-dhph"
server_key: "fac596657fd7dbd61606c281e0b7e2ae"
server_key_iv: "21fe137ea116d2128b14d73a39516e13"
server_key_version: "encryptionkey@0"
I20260806 02:58:32.121657 32598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:58:32.142234 32598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:58:32.147138 32598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:58:32.147461 32598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:58:32.168995 32598 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.254:45541
I20260806 02:58:32.168998 32666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.254:45541 every 8 connection(s)
I20260806 02:58:32.170490 32598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
I20260806 02:58:32.174204 32667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:58:32.179335 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 32598
I20260806 02:58:32.179592 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/minicluster-data/master-0/wal/instance
I20260806 02:58:32.180383 32043 external_mini_cluster.cc:1428] Setting key d0efbc4f55fdf1fc3c2ce8abca9dc884
I20260806 02:58:32.187202 32667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42f81ed814ed4066a5071cd2c685c8e2: Bootstrap starting.
I20260806 02:58:32.191183 32667 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42f81ed814ed4066a5071cd2c685c8e2: Neither blocks nor log segments found. Creating new log.
I20260806 02:58:32.192523 32667 log.cc:824] T 00000000000000000000000000000000 P 42f81ed814ed4066a5071cd2c685c8e2: Log is configured to *not* fsync() on all Append() calls
I20260806 02:58:32.197005 32667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42f81ed814ed4066a5071cd2c685c8e2: No bootstrap required, opened a new log
Aug 06 02:58:32 dist-test-slave-dhph krb5kdc[32585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 02:58:32.206888 32667 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42f81ed814ed4066a5071cd2c685c8e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42f81ed814ed4066a5071cd2c685c8e2" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 45541 } }
I20260806 02:58:32.207716 32667 raft_consensus.cc:374] T 00000000000000000000000000000000 P 42f81ed814ed4066a5071cd2c685c8e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:58:32.207918 32667 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42f81ed814ed4066a5071cd2c685c8e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42f81ed814ed4066a5071cd2c685c8e2, State: Initialized, Role: FOLLOWER
I20260806 02:58:32.208343 32670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:32.183068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33034 (local address 127.31.74.254:45541)
0806 02:58:32.184156 (+ 1088us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:32.184170 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:32.184224 (+ 54us) server_negotiation.cc:407] Connection header received
0806 02:58:32.186204 (+ 1980us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:32.186230 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:32.186748 (+ 518us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:32.187187 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:32.189121 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:32.190823 (+ 1702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:32.192205 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:32.192984 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:32.193226 (+ 242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:32.199120 (+ 5894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:58:32.199133 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:58:32.199153 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:58:32.199199 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:58:32.202760 (+ 3561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:32.203676 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:32.203683 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:32.203694 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:32.203904 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:32.204350 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:32.204354 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:32.204356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:32.205094 (+ 738us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:58:32.205332 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:32.206372 (+ 1040us) server_negotiation.cc:299] Negotiation successful
0806 02:58:32.207056 (+ 684us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":687,"thread_start_us":416,"threads_started":1}
I20260806 02:58:32.208666 32667 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42f81ed814ed4066a5071cd2c685c8e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42f81ed814ed4066a5071cd2c685c8e2" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 45541 } }
I20260806 02:58:32.208979 32667 raft_consensus.cc:388] T 00000000000000000000000000000000 P 42f81ed814ed4066a5071cd2c685c8e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:58:32.209156 32667 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42f81ed814ed4066a5071cd2c685c8e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:58:32.209357 32667 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42f81ed814ed4066a5071cd2c685c8e2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:58:32.213370 32667 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42f81ed814ed4066a5071cd2c685c8e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42f81ed814ed4066a5071cd2c685c8e2" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 45541 } }
I20260806 02:58:32.213989 32667 leader_election.cc:302] T 00000000000000000000000000000000 P 42f81ed814ed4066a5071cd2c685c8e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 42f81ed814ed4066a5071cd2c685c8e2; no voters:
I20260806 02:58:32.214915 32667 leader_election.cc:288] T 00000000000000000000000000000000 P 42f81ed814ed4066a5071cd2c685c8e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:58:32.215080 32672 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 42f81ed814ed4066a5071cd2c685c8e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:58:32.216602 32672 raft_consensus.cc:686] T 00000000000000000000000000000000 P 42f81ed814ed4066a5071cd2c685c8e2 [term 1 LEADER]: Becoming Leader. State: Replica: 42f81ed814ed4066a5071cd2c685c8e2, State: Running, Role: LEADER
I20260806 02:58:32.217190 32672 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42f81ed814ed4066a5071cd2c685c8e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42f81ed814ed4066a5071cd2c685c8e2" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 45541 } }
I20260806 02:58:32.217413 32667 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42f81ed814ed4066a5071cd2c685c8e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:58:32.220986 32673 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42f81ed814ed4066a5071cd2c685c8e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "42f81ed814ed4066a5071cd2c685c8e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42f81ed814ed4066a5071cd2c685c8e2" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 45541 } } }
I20260806 02:58:32.221592 32673 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42f81ed814ed4066a5071cd2c685c8e2 [sys.catalog]: This master's current role is: LEADER
I20260806 02:58:32.221477 32674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42f81ed814ed4066a5071cd2c685c8e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42f81ed814ed4066a5071cd2c685c8e2. Latest consensus state: current_term: 1 leader_uuid: "42f81ed814ed4066a5071cd2c685c8e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42f81ed814ed4066a5071cd2c685c8e2" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 45541 } } }
I20260806 02:58:32.221787 32674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42f81ed814ed4066a5071cd2c685c8e2 [sys.catalog]: This master's current role is: LEADER
I20260806 02:58:32.222826 32678 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:58:32.229367 32678 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:58:32.242654 32678 catalog_manager.cc:1347] Generated new cluster ID: 5a6c57f6158543b4a9bea2ce7af64e6a
I20260806 02:58:32.242810 32678 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:58:32.257169 32678 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:58:32.258878 32678 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:58:32.267360 32678 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 42f81ed814ed4066a5071cd2c685c8e2: Generated new TSK 0
I20260806 02:58:32.268299 32678 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.74.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.193.keytab.
Entry for principal kudu/127.31.74.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.193.keytab.
I20260806 02:58:32.341816 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.193:0
--local_ip_for_outbound_sockets=127.31.74.193
--webserver_interface=127.31.74.193
--webserver_port=0
--tserver_master_addrs=127.31.74.254:45541
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.193.keytab
--principal=kudu/127.31.74.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:46179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:58:32.782001 32693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:58:32.782312 32693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:58:32.782480 32693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:58:32.791517 32693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:58:32.791669 32693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:58:32.791899 32693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.193
I20260806 02:58:32.804708 32693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:46179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.193.keytab
--principal=kudu/127.31.74.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.74.193
--webserver_port=0
--tserver_master_addrs=127.31.74.254:45541
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.193
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:58:32.806231 32693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:58:32.807843 32693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:58:32.823871 32701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:58:32.823968 32699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:58:32.823907 32698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:58:32.825491 32693 server_base.cc:1034] running on GCE node
I20260806 02:58:32.826300 32693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:58:32.827601 32693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:58:32.828876 32693 hybrid_clock.cc:648] HybridClock initialized: now 1785985112828805 us; error 64 us; skew 500 ppm
I20260806 02:58:32.829288 32693 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:58:32 dist-test-slave-dhph krb5kdc[32585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985112, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:58:32.832964 32693 init.cc:377] Logged in from keytab as kudu/127.31.74.193@KRBTEST.COM (short username kudu)
I20260806 02:58:32.834946 32693 webserver.cc:460] Webserver started at http://127.31.74.193:46839/ using document root <none> and password file <none>
I20260806 02:58:32.835768 32693 fs_manager.cc:359] Metadata directory not provided
I20260806 02:58:32.835951 32693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:58:32.836589 32693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:58:32.840353 32693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/minicluster-data/ts-0/data/instance:
uuid: "067a1b0f46d248bb90f55d942131fb20"
format_stamp: "Formatted at 2026-08-06 02:58:32 on dist-test-slave-dhph"
server_key: "816fdab51e70b3b8730c4653c54bcf8a"
server_key_iv: "5ec33086c4e6c767eeb551ce70a41e67"
server_key_version: "encryptionkey@0"
I20260806 02:58:32.841728 32693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/minicluster-data/ts-0/wal/instance:
uuid: "067a1b0f46d248bb90f55d942131fb20"
format_stamp: "Formatted at 2026-08-06 02:58:32 on dist-test-slave-dhph"
server_key: "816fdab51e70b3b8730c4653c54bcf8a"
server_key_iv: "5ec33086c4e6c767eeb551ce70a41e67"
server_key_version: "encryptionkey@0"
I20260806 02:58:32.848277 32693 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260806 02:58:32.852378 32708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:58:32.854002 32693 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260806 02:58:32.854228 32693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/minicluster-data/ts-0/wal
uuid: "067a1b0f46d248bb90f55d942131fb20"
format_stamp: "Formatted at 2026-08-06 02:58:32 on dist-test-slave-dhph"
server_key: "816fdab51e70b3b8730c4653c54bcf8a"
server_key_iv: "5ec33086c4e6c767eeb551ce70a41e67"
server_key_version: "encryptionkey@0"
I20260806 02:58:32.854481 32693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:58:32.883194 32693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:58:32.889703 32693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:58:32.890110 32693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:58:32.891770 32693 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:58:32.893594 32693 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:58:32.893730 32693 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:58:32.893908 32693 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:58:32.893994 32693 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:58:32.942076 32693 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.193:39187
I20260806 02:58:32.942377 354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.193:39187 every 8 connection(s)
I20260806 02:58:32.943562 32693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
I20260806 02:58:32.951570 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 32693
I20260806 02:58:32.951848 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/minicluster-data/ts-0/wal/instance
I20260806 02:58:32.952524 32043 external_mini_cluster.cc:1428] Setting key ab45f09f345a999259266c79ef61e5a0
Aug 06 02:58:32 dist-test-slave-dhph krb5kdc[32585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985112, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.193@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 02:58:32.971621 357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:32.946923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.193:46677 (local address 127.31.74.254:45541)
0806 02:58:32.947480 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:32.947487 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:32.948632 (+ 1145us) server_negotiation.cc:407] Connection header received
0806 02:58:32.950733 (+ 2101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:32.950740 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:32.950828 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:32.950941 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:32.952984 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:32.953970 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:32.958098 (+ 4128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:32.959066 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:32.959267 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:32.964069 (+ 4802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:58:32.964075 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:58:32.964079 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:58:32.964120 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:58:32.967179 (+ 3059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:32.968306 (+ 1127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:32.968315 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:32.968321 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:32.968490 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:32.969156 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:32.969164 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:32.969175 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:32.969492 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:58:32.969620 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:32.970968 (+ 1348us) server_negotiation.cc:299] Negotiation successful
0806 02:58:32.971265 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":254,"threads_started":1}
I20260806 02:58:32.973702 358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:32.947631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:45541 (local address 127.31.74.193:46677)
0806 02:58:32.948458 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:32.948503 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 02:58:32.950402 (+ 1899us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:58:32.951283 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:58:32.951291 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:58:32.951994 (+ 703us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:58:32.952609 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:32.952657 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:32.954182 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:32.954192 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:32.956901 (+ 2709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:32.956924 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:32.959413 (+ 2489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:32.959422 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:32.959726 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:32.960585 (+ 859us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:58:32.960619 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:58:32.963477 (+ 2858us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:58:32.967374 (+ 3897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:58:32.967385 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:58:32.967406 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:58:32.968101 (+ 695us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:58:32.968625 (+ 524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:58:32.968633 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:58:32.968638 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:58:32.968981 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:58:32.969692 (+ 711us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:58:32.969704 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:58:32.970401 (+ 697us) client_negotiation.cc:769] Sending connection context
0806 02:58:32.970819 (+ 418us) client_negotiation.cc:240] Negotiation successful
0806 02:58:32.971480 (+ 661us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":343,"threads_started":1}
I20260806 02:58:32.975811 355 heartbeater.cc:347] Connected to a master server at 127.31.74.254:45541
WARNING: no policy specified for kudu/127.31.74.194@KRBTEST.COM; defaulting to no policy
I20260806 02:58:32.976194 355 heartbeater.cc:464] Registering TS with master...
I20260806 02:58:32.976970 355 heartbeater.cc:511] Master 127.31.74.254:45541 requested a full tablet report, sending...
I20260806 02:58:32.979404 32630 ts_manager.cc:194] Registered new tserver with Master: 067a1b0f46d248bb90f55d942131fb20 (127.31.74.193:39187)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.194@KRBTEST.COM" created.
I20260806 02:58:32.981514 32630 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.74.193@KRBTEST.COM'} at 127.31.74.193:46677
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.194.keytab.
Entry for principal kudu/127.31.74.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.194.keytab.
I20260806 02:58:33.012646 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.194:0
--local_ip_for_outbound_sockets=127.31.74.194
--webserver_interface=127.31.74.194
--webserver_port=0
--tserver_master_addrs=127.31.74.254:45541
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.194.keytab
--principal=kudu/127.31.74.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:46179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:58:33.400024 361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:58:33.400346 361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:58:33.400517 361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:58:33.408885 361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:58:33.409029 361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:58:33.409278 361 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.194
I20260806 02:58:33.421097 361 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:46179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.194.keytab
--principal=kudu/127.31.74.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.74.194
--webserver_port=0
--tserver_master_addrs=127.31.74.254:45541
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.194
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:58:33.422494 361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:58:33.423960 361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:58:33.439411 367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:58:33.439386 369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:58:33.439796 366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:58:33.441162 361 server_base.cc:1034] running on GCE node
I20260806 02:58:33.441882 361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:58:33.443096 361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:58:33.444332 361 hybrid_clock.cc:648] HybridClock initialized: now 1785985113444282 us; error 37 us; skew 500 ppm
I20260806 02:58:33.444639 361 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:58:33 dist-test-slave-dhph krb5kdc[32585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985113, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:58:33.448283 361 init.cc:377] Logged in from keytab as kudu/127.31.74.194@KRBTEST.COM (short username kudu)
I20260806 02:58:33.450037 361 webserver.cc:460] Webserver started at http://127.31.74.194:33439/ using document root <none> and password file <none>
I20260806 02:58:33.450861 361 fs_manager.cc:359] Metadata directory not provided
I20260806 02:58:33.451001 361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:58:33.451354 361 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:58:33.454716 361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/minicluster-data/ts-1/data/instance:
uuid: "d29a20e5614b4436a6f63810e9d045e8"
format_stamp: "Formatted at 2026-08-06 02:58:33 on dist-test-slave-dhph"
server_key: "c290041bb2e6376515231e48d4e49641"
server_key_iv: "e0d70457d760e6224213f4e4e9cc4555"
server_key_version: "encryptionkey@0"
I20260806 02:58:33.455703 361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/minicluster-data/ts-1/wal/instance:
uuid: "d29a20e5614b4436a6f63810e9d045e8"
format_stamp: "Formatted at 2026-08-06 02:58:33 on dist-test-slave-dhph"
server_key: "c290041bb2e6376515231e48d4e49641"
server_key_iv: "e0d70457d760e6224213f4e4e9cc4555"
server_key_version: "encryptionkey@0"
I20260806 02:58:33.462891 361 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260806 02:58:33.467199 376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:58:33.469064 361 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 02:58:33.469339 361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/minicluster-data/ts-1/wal
uuid: "d29a20e5614b4436a6f63810e9d045e8"
format_stamp: "Formatted at 2026-08-06 02:58:33 on dist-test-slave-dhph"
server_key: "c290041bb2e6376515231e48d4e49641"
server_key_iv: "e0d70457d760e6224213f4e4e9cc4555"
server_key_version: "encryptionkey@0"
I20260806 02:58:33.469627 361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:58:33.500352 361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:58:33.506726 361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:58:33.507079 361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:58:33.508266 361 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:58:33.509855 361 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:58:33.509968 361 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:58:33.510133 361 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:58:33.510210 361 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:58:33.551913 361 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.194:38425
I20260806 02:58:33.551949 490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.194:38425 every 8 connection(s)
I20260806 02:58:33.553368 361 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
I20260806 02:58:33.559187 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 361
I20260806 02:58:33.559405 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/minicluster-data/ts-1/wal/instance
I20260806 02:58:33.559886 32043 external_mini_cluster.cc:1428] Setting key e8ba2e3198cc1d4f3f093462fecebc6b
Aug 06 02:58:33 dist-test-slave-dhph krb5kdc[32585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985113, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.194@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 02:58:33.580107 493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:33.556495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.194:49259 (local address 127.31.74.254:45541)
0806 02:58:33.556986 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:33.556990 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:33.558312 (+ 1322us) server_negotiation.cc:407] Connection header received
0806 02:58:33.560958 (+ 2646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:33.560963 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:33.561043 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:33.561157 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:33.563064 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:33.564580 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:33.566620 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:33.567521 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:33.567712 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:33.572969 (+ 5257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:58:33.572974 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:58:33.572977 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:58:33.573019 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:58:33.576647 (+ 3628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:33.577536 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:33.577541 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:33.577543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:33.577669 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:33.578154 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:33.578159 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:33.578161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:33.578471 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:58:33.578660 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:33.579426 (+ 766us) server_negotiation.cc:299] Negotiation successful
0806 02:58:33.579676 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":260,"threads_started":1}
I20260806 02:58:33.581633 494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:33.557099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:45541 (local address 127.31.74.194:49259)
0806 02:58:33.558109 (+ 1010us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:33.558159 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 02:58:33.560583 (+ 2424us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:58:33.561454 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:58:33.561467 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:58:33.562153 (+ 686us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:58:33.562865 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:33.562887 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:33.564791 (+ 1904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:33.564803 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:33.566335 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:33.566366 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:33.567823 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:33.567829 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:33.568069 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:33.568988 (+ 919us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:58:33.569015 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:58:33.572658 (+ 3643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:58:33.576826 (+ 4168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:58:33.576837 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:58:33.576863 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:58:33.577376 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:58:33.577793 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:58:33.577798 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:58:33.577802 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:58:33.578014 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:58:33.578636 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:58:33.578645 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:58:33.579037 (+ 392us) client_negotiation.cc:769] Sending connection context
0806 02:58:33.579335 (+ 298us) client_negotiation.cc:240] Negotiation successful
0806 02:58:33.579931 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":761,"thread_start_us":510,"threads_started":1}
I20260806 02:58:33.583611 491 heartbeater.cc:347] Connected to a master server at 127.31.74.254:45541
I20260806 02:58:33.583967 491 heartbeater.cc:464] Registering TS with master...
I20260806 02:58:33.584775 491 heartbeater.cc:511] Master 127.31.74.254:45541 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.31.74.195@KRBTEST.COM; defaulting to no policy
I20260806 02:58:33.586633 32630 ts_manager.cc:194] Registered new tserver with Master: d29a20e5614b4436a6f63810e9d045e8 (127.31.74.194:38425)
I20260806 02:58:33.587718 32630 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.74.194@KRBTEST.COM'} at 127.31.74.194:49259
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.195.keytab.
Entry for principal kudu/127.31.74.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.195.keytab.
I20260806 02:58:33.623764 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.195:0
--local_ip_for_outbound_sockets=127.31.74.195
--webserver_interface=127.31.74.195
--webserver_port=0
--tserver_master_addrs=127.31.74.254:45541
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.195.keytab
--principal=kudu/127.31.74.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:46179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:58:33.985234 355 heartbeater.cc:503] Master 127.31.74.254:45541 was elected leader, sending a full tablet report...
W20260806 02:58:34.023658 497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:58:34.024042 497 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:58:34.024217 497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:58:34.032352 497 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:58:34.032572 497 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:58:34.032804 497 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.195
I20260806 02:58:34.045204 497 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:46179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.195.keytab
--principal=kudu/127.31.74.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.74.195
--webserver_port=0
--tserver_master_addrs=127.31.74.254:45541
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.195
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:58:34.047230 497 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:58:34.048827 497 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:58:34.066622 502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:58:34.066769 503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:58:34.066752 505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:58:34.068588 497 server_base.cc:1034] running on GCE node
I20260806 02:58:34.069685 497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:58:34.070956 497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:58:34.072233 497 hybrid_clock.cc:648] HybridClock initialized: now 1785985114072168 us; error 66 us; skew 500 ppm
I20260806 02:58:34.072610 497 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:58:34 dist-test-slave-dhph krb5kdc[32585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985114, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:58:34.076467 497 init.cc:377] Logged in from keytab as kudu/127.31.74.195@KRBTEST.COM (short username kudu)
I20260806 02:58:34.078567 497 webserver.cc:460] Webserver started at http://127.31.74.195:38805/ using document root <none> and password file <none>
I20260806 02:58:34.079437 497 fs_manager.cc:359] Metadata directory not provided
I20260806 02:58:34.079609 497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:58:34.079977 497 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:58:34.083499 497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/minicluster-data/ts-2/data/instance:
uuid: "93677f8a09994560884d95d0948cd44b"
format_stamp: "Formatted at 2026-08-06 02:58:34 on dist-test-slave-dhph"
server_key: "4f708a56403a5064949505a1c9e0abce"
server_key_iv: "e4d71b8ecff0706cc62b1963e11f48a0"
server_key_version: "encryptionkey@0"
I20260806 02:58:34.084467 497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/minicluster-data/ts-2/wal/instance:
uuid: "93677f8a09994560884d95d0948cd44b"
format_stamp: "Formatted at 2026-08-06 02:58:34 on dist-test-slave-dhph"
server_key: "4f708a56403a5064949505a1c9e0abce"
server_key_iv: "e4d71b8ecff0706cc62b1963e11f48a0"
server_key_version: "encryptionkey@0"
I20260806 02:58:34.091109 497 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260806 02:58:34.096539 513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:58:34.098409 497 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.001s
I20260806 02:58:34.098675 497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/minicluster-data/ts-2/wal
uuid: "93677f8a09994560884d95d0948cd44b"
format_stamp: "Formatted at 2026-08-06 02:58:34 on dist-test-slave-dhph"
server_key: "4f708a56403a5064949505a1c9e0abce"
server_key_iv: "e4d71b8ecff0706cc62b1963e11f48a0"
server_key_version: "encryptionkey@0"
I20260806 02:58:34.098940 497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:58:34.122710 497 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:58:34.128816 497 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:58:34.129199 497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:58:34.130513 497 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:58:34.132360 497 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:58:34.132493 497 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:58:34.132668 497 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:58:34.132766 497 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:58:34.183696 497 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.195:33659
I20260806 02:58:34.183763 629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.195:33659 every 8 connection(s)
I20260806 02:58:34.185216 497 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
I20260806 02:58:34.193485 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 497
I20260806 02:58:34.193789 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785985105255475-32043-0/minicluster-data/ts-2/wal/instance
I20260806 02:58:34.194283 32043 external_mini_cluster.cc:1428] Setting key 655aa07c6a107a4ebebf2f8be3ca81e4
Aug 06 02:58:34 dist-test-slave-dhph krb5kdc[32585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985114, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.195@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 02:58:34.211930 632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:34.188346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.195:52427 (local address 127.31.74.254:45541)
0806 02:58:34.188929 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:34.188934 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:34.190140 (+ 1206us) server_negotiation.cc:407] Connection header received
0806 02:58:34.192814 (+ 2674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:34.192821 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:34.192961 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:34.193100 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:34.195236 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:34.196579 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:34.199086 (+ 2507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:34.199541 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:34.199664 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:34.203896 (+ 4232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:58:34.203901 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:58:34.203905 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:58:34.203944 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:58:34.207309 (+ 3365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:34.208258 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:34.208263 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:34.208266 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:34.208408 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:34.209082 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:34.209087 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:34.209091 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:34.209458 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:58:34.209621 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:34.211097 (+ 1476us) server_negotiation.cc:299] Negotiation successful
0806 02:58:34.211324 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":294,"threads_started":1}
I20260806 02:58:34.212949 633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:34.189018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:45541 (local address 127.31.74.195:52427)
0806 02:58:34.189942 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:34.189999 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0806 02:58:34.192461 (+ 2462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:58:34.193397 (+ 936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:58:34.193407 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:58:34.194132 (+ 725us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:58:34.194949 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:34.194978 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:34.196785 (+ 1807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:34.196789 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:34.198852 (+ 2063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:34.198866 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:34.199839 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:34.199846 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:34.200063 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:34.200846 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:58:34.200876 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:58:34.203583 (+ 2707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:58:34.207497 (+ 3914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:58:34.207505 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:58:34.207521 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:58:34.208058 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:58:34.208557 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:58:34.208562 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:58:34.208564 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:58:34.208886 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:58:34.209631 (+ 745us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:58:34.209638 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:58:34.210459 (+ 821us) client_negotiation.cc:769] Sending connection context
0806 02:58:34.210910 (+ 451us) client_negotiation.cc:240] Negotiation successful
0806 02:58:34.211556 (+ 646us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":398,"threads_started":1}
I20260806 02:58:34.216675 630 heartbeater.cc:347] Connected to a master server at 127.31.74.254:45541
I20260806 02:58:34.217072 630 heartbeater.cc:464] Registering TS with master...
I20260806 02:58:34.218044 630 heartbeater.cc:511] Master 127.31.74.254:45541 requested a full tablet report, sending...
I20260806 02:58:34.220964 32630 ts_manager.cc:194] Registered new tserver with Master: 93677f8a09994560884d95d0948cd44b (127.31.74.195:33659)
I20260806 02:58:34.222291 32630 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.74.195@KRBTEST.COM'} at 127.31.74.195:52427
I20260806 02:58:34.225028 32043 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:58:34.250046 632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:34.233529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33044 (local address 127.31.74.254:45541)
0806 02:58:34.233868 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:34.233878 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:34.234115 (+ 237us) server_negotiation.cc:407] Connection header received
0806 02:58:34.234299 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:34.234307 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:34.234715 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:34.234885 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:34.236906 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:34.237785 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:34.239361 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:34.239859 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:34.239983 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:34.242640 (+ 2657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:58:34.242647 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:58:34.242651 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:58:34.242719 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:58:34.246171 (+ 3452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:34.247546 (+ 1375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:34.247552 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:34.247554 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:34.247796 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:34.248344 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:34.248348 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:34.248350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:34.248743 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:58:34.249006 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:34.249472 (+ 466us) server_negotiation.cc:299] Negotiation successful
0806 02:58:34.249758 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140}
I20260806 02:58:34.257920 32630 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33044:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:58:34.265245 32630 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:58:34.298111 643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:34.279802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.194:38425 (local address 127.0.0.1:52114)
0806 02:58:34.282374 (+ 2572us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:34.282446 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0806 02:58:34.282817 (+ 371us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:58:34.284053 (+ 1236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:58:34.284065 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:58:34.284123 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:58:34.284731 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:34.284757 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:34.287629 (+ 2872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:34.287639 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:34.293346 (+ 5707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:34.293372 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:34.296902 (+ 3530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:34.296910 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:34.297205 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:34.297256 (+ 51us) client_negotiation.cc:769] Sending connection context
0806 02:58:34.297416 (+ 160us) client_negotiation.cc:240] Negotiation successful
0806 02:58:34.297637 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2224,"thread_start_us":2027,"threads_started":1}
I20260806 02:58:34.298032 644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:34.280063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52114 (local address 127.31.74.194:38425)
0806 02:58:34.281612 (+ 1549us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:34.281627 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:34.282604 (+ 977us) server_negotiation.cc:407] Connection header received
0806 02:58:34.283149 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:34.283163 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:34.283484 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:34.283669 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:58:34.285004 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:34.287365 (+ 2361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:34.293797 (+ 6432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:34.296504 (+ 2707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:34.296702 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:34.297011 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:34.297465 (+ 454us) server_negotiation.cc:299] Negotiation successful
0806 02:58:34.297654 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1308,"thread_start_us":1175,"threads_started":1}
I20260806 02:58:34.300539 645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:34.280264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.195:33659 (local address 127.0.0.1:35092)
0806 02:58:34.283535 (+ 3271us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:34.283579 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 02:58:34.283718 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:58:34.286508 (+ 2790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:58:34.286516 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:58:34.286556 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:58:34.287185 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:34.287202 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:34.291874 (+ 4672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:34.291885 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:34.296995 (+ 5110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:34.297016 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:34.299615 (+ 2599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:34.299623 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:34.299839 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:34.299877 (+ 38us) client_negotiation.cc:769] Sending connection context
0806 02:58:34.299971 (+ 94us) client_negotiation.cc:240] Negotiation successful
0806 02:58:34.300122 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3173,"thread_start_us":3013,"threads_started":1}
I20260806 02:58:34.301491 646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:34.280961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35092 (local address 127.31.74.195:33659)
0806 02:58:34.283305 (+ 2344us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:34.283316 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:34.283886 (+ 570us) server_negotiation.cc:407] Connection header received
0806 02:58:34.285538 (+ 1652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:34.285551 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:34.285874 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:34.286110 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:58:34.287485 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:34.291600 (+ 4115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:34.297212 (+ 5612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:34.299069 (+ 1857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:34.299216 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:34.299465 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:34.300106 (+ 641us) server_negotiation.cc:299] Negotiation successful
0806 02:58:34.301192 (+ 1086us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2129,"thread_start_us":1939,"threads_started":1}
I20260806 02:58:34.303926 425 tablet_service.cc:1467] Processing CreateTablet for tablet d78670f612f344a18730c5c3c2ae38f3 (DEFAULT_TABLE table=test-table [id=4dee5784fc7647d4b49e578b6923d158]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:58:34.305351 562 tablet_service.cc:1467] Processing CreateTablet for tablet d78670f612f344a18730c5c3c2ae38f3 (DEFAULT_TABLE table=test-table [id=4dee5784fc7647d4b49e578b6923d158]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:58:34.306470 647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:34.283668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:39187 (local address 127.0.0.1:57048)
0806 02:58:34.295493 (+ 11825us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:34.295518 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 02:58:34.295660 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:58:34.297684 (+ 2024us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:58:34.297690 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:58:34.297725 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:58:34.298163 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:34.298176 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:34.300564 (+ 2388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:34.300570 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:34.303109 (+ 2539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:34.303131 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:34.305573 (+ 2442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:34.305579 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:34.305757 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:34.305796 (+ 39us) client_negotiation.cc:769] Sending connection context
0806 02:58:34.305900 (+ 104us) client_negotiation.cc:240] Negotiation successful
0806 02:58:34.306076 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11665,"thread_start_us":12221,"threads_started":1}
I20260806 02:58:34.306754 425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d78670f612f344a18730c5c3c2ae38f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:58:34.306974 562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d78670f612f344a18730c5c3c2ae38f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:58:34.307301 648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:34.284071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57048 (local address 127.31.74.193:39187)
0806 02:58:34.296864 (+ 12793us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:34.296875 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:34.296907 (+ 32us) server_negotiation.cc:407] Connection header received
0806 02:58:34.297029 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:34.297040 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:34.297379 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:34.298412 (+ 1033us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:58:34.298682 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:34.300386 (+ 1704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:34.303340 (+ 2954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:34.305249 (+ 1909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:34.305426 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:34.305683 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:34.306776 (+ 1093us) server_negotiation.cc:299] Negotiation successful
0806 02:58:34.306956 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12593,"thread_start_us":12380,"threads_started":1}
I20260806 02:58:34.311642 32756 tablet_service.cc:1467] Processing CreateTablet for tablet d78670f612f344a18730c5c3c2ae38f3 (DEFAULT_TABLE table=test-table [id=4dee5784fc7647d4b49e578b6923d158]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:58:34.313410 32756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d78670f612f344a18730c5c3c2ae38f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:58:34.325785 649 tablet_bootstrap.cc:492] T d78670f612f344a18730c5c3c2ae38f3 P 93677f8a09994560884d95d0948cd44b: Bootstrap starting.
I20260806 02:58:34.328883 650 tablet_bootstrap.cc:492] T d78670f612f344a18730c5c3c2ae38f3 P d29a20e5614b4436a6f63810e9d045e8: Bootstrap starting.
I20260806 02:58:34.331197 649 tablet_bootstrap.cc:654] T d78670f612f344a18730c5c3c2ae38f3 P 93677f8a09994560884d95d0948cd44b: Neither blocks nor log segments found. Creating new log.
I20260806 02:58:34.332813 649 log.cc:824] T d78670f612f344a18730c5c3c2ae38f3 P 93677f8a09994560884d95d0948cd44b: Log is configured to *not* fsync() on all Append() calls
I20260806 02:58:34.333171 650 tablet_bootstrap.cc:654] T d78670f612f344a18730c5c3c2ae38f3 P d29a20e5614b4436a6f63810e9d045e8: Neither blocks nor log segments found. Creating new log.
I20260806 02:58:34.333652 651 tablet_bootstrap.cc:492] T d78670f612f344a18730c5c3c2ae38f3 P 067a1b0f46d248bb90f55d942131fb20: Bootstrap starting.
I20260806 02:58:34.335140 650 log.cc:824] T d78670f612f344a18730c5c3c2ae38f3 P d29a20e5614b4436a6f63810e9d045e8: Log is configured to *not* fsync() on all Append() calls
I20260806 02:58:34.337393 651 tablet_bootstrap.cc:654] T d78670f612f344a18730c5c3c2ae38f3 P 067a1b0f46d248bb90f55d942131fb20: Neither blocks nor log segments found. Creating new log.
I20260806 02:58:34.339147 651 log.cc:824] T d78670f612f344a18730c5c3c2ae38f3 P 067a1b0f46d248bb90f55d942131fb20: Log is configured to *not* fsync() on all Append() calls
I20260806 02:58:34.342933 649 tablet_bootstrap.cc:492] T d78670f612f344a18730c5c3c2ae38f3 P 93677f8a09994560884d95d0948cd44b: No bootstrap required, opened a new log
I20260806 02:58:34.343360 649 ts_tablet_manager.cc:1397] T d78670f612f344a18730c5c3c2ae38f3 P 93677f8a09994560884d95d0948cd44b: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.007s
I20260806 02:58:34.343905 651 tablet_bootstrap.cc:492] T d78670f612f344a18730c5c3c2ae38f3 P 067a1b0f46d248bb90f55d942131fb20: No bootstrap required, opened a new log
I20260806 02:58:34.344204 650 tablet_bootstrap.cc:492] T d78670f612f344a18730c5c3c2ae38f3 P d29a20e5614b4436a6f63810e9d045e8: No bootstrap required, opened a new log
I20260806 02:58:34.344513 651 ts_tablet_manager.cc:1397] T d78670f612f344a18730c5c3c2ae38f3 P 067a1b0f46d248bb90f55d942131fb20: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20260806 02:58:34.344522 650 ts_tablet_manager.cc:1397] T d78670f612f344a18730c5c3c2ae38f3 P d29a20e5614b4436a6f63810e9d045e8: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.004s
I20260806 02:58:34.354167 651 raft_consensus.cc:348] T d78670f612f344a18730c5c3c2ae38f3 P 067a1b0f46d248bb90f55d942131fb20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d29a20e5614b4436a6f63810e9d045e8" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 38425 } } peers { permanent_uuid: "93677f8a09994560884d95d0948cd44b" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 33659 } } peers { permanent_uuid: "067a1b0f46d248bb90f55d942131fb20" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 39187 } }
I20260806 02:58:34.354924 651 raft_consensus.cc:374] T d78670f612f344a18730c5c3c2ae38f3 P 067a1b0f46d248bb90f55d942131fb20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:58:34.355072 651 raft_consensus.cc:729] T d78670f612f344a18730c5c3c2ae38f3 P 067a1b0f46d248bb90f55d942131fb20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 067a1b0f46d248bb90f55d942131fb20, State: Initialized, Role: FOLLOWER
I20260806 02:58:34.356042 651 consensus_queue.cc:260] T d78670f612f344a18730c5c3c2ae38f3 P 067a1b0f46d248bb90f55d942131fb20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d29a20e5614b4436a6f63810e9d045e8" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 38425 } } peers { permanent_uuid: "93677f8a09994560884d95d0948cd44b" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 33659 } } peers { permanent_uuid: "067a1b0f46d248bb90f55d942131fb20" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 39187 } }
I20260806 02:58:34.357116 649 raft_consensus.cc:348] T d78670f612f344a18730c5c3c2ae38f3 P 93677f8a09994560884d95d0948cd44b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d29a20e5614b4436a6f63810e9d045e8" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 38425 } } peers { permanent_uuid: "93677f8a09994560884d95d0948cd44b" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 33659 } } peers { permanent_uuid: "067a1b0f46d248bb90f55d942131fb20" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 39187 } }
I20260806 02:58:34.357774 649 raft_consensus.cc:374] T d78670f612f344a18730c5c3c2ae38f3 P 93677f8a09994560884d95d0948cd44b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:58:34.357906 649 raft_consensus.cc:729] T d78670f612f344a18730c5c3c2ae38f3 P 93677f8a09994560884d95d0948cd44b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93677f8a09994560884d95d0948cd44b, State: Initialized, Role: FOLLOWER
I20260806 02:58:34.358888 651 ts_tablet_manager.cc:1428] T d78670f612f344a18730c5c3c2ae38f3 P 067a1b0f46d248bb90f55d942131fb20: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260806 02:58:34.358721 649 consensus_queue.cc:260] T d78670f612f344a18730c5c3c2ae38f3 P 93677f8a09994560884d95d0948cd44b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d29a20e5614b4436a6f63810e9d045e8" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 38425 } } peers { permanent_uuid: "93677f8a09994560884d95d0948cd44b" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 33659 } } peers { permanent_uuid: "067a1b0f46d248bb90f55d942131fb20" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 39187 } }
I20260806 02:58:34.359508 650 raft_consensus.cc:348] T d78670f612f344a18730c5c3c2ae38f3 P d29a20e5614b4436a6f63810e9d045e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d29a20e5614b4436a6f63810e9d045e8" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 38425 } } peers { permanent_uuid: "93677f8a09994560884d95d0948cd44b" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 33659 } } peers { permanent_uuid: "067a1b0f46d248bb90f55d942131fb20" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 39187 } }
I20260806 02:58:34.360090 630 heartbeater.cc:503] Master 127.31.74.254:45541 was elected leader, sending a full tablet report...
I20260806 02:58:34.360137 650 raft_consensus.cc:374] T d78670f612f344a18730c5c3c2ae38f3 P d29a20e5614b4436a6f63810e9d045e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:58:34.360255 650 raft_consensus.cc:729] T d78670f612f344a18730c5c3c2ae38f3 P d29a20e5614b4436a6f63810e9d045e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d29a20e5614b4436a6f63810e9d045e8, State: Initialized, Role: FOLLOWER
I20260806 02:58:34.361089 649 ts_tablet_manager.cc:1428] T d78670f612f344a18730c5c3c2ae38f3 P 93677f8a09994560884d95d0948cd44b: Time spent starting tablet: real 0.018s user 0.016s sys 0.001s
I20260806 02:58:34.361090 650 consensus_queue.cc:260] T d78670f612f344a18730c5c3c2ae38f3 P d29a20e5614b4436a6f63810e9d045e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d29a20e5614b4436a6f63810e9d045e8" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 38425 } } peers { permanent_uuid: "93677f8a09994560884d95d0948cd44b" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 33659 } } peers { permanent_uuid: "067a1b0f46d248bb90f55d942131fb20" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 39187 } }
I20260806 02:58:34.362512 491 heartbeater.cc:503] Master 127.31.74.254:45541 was elected leader, sending a full tablet report...
I20260806 02:58:34.364122 650 ts_tablet_manager.cc:1428] T d78670f612f344a18730c5c3c2ae38f3 P d29a20e5614b4436a6f63810e9d045e8: Time spent starting tablet: real 0.019s user 0.018s sys 0.000s
W20260806 02:58:34.437242 631 tablet.cc:2369] T d78670f612f344a18730c5c3c2ae38f3 P 93677f8a09994560884d95d0948cd44b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:58:34.446947 356 tablet.cc:2369] T d78670f612f344a18730c5c3c2ae38f3 P 067a1b0f46d248bb90f55d942131fb20: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:58:34.555835 492 tablet.cc:2369] T d78670f612f344a18730c5c3c2ae38f3 P d29a20e5614b4436a6f63810e9d045e8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:58:34.635612 655 raft_consensus.cc:482] T d78670f612f344a18730c5c3c2ae38f3 P 067a1b0f46d248bb90f55d942131fb20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:58:34.636713 655 raft_consensus.cc:504] T d78670f612f344a18730c5c3c2ae38f3 P 067a1b0f46d248bb90f55d942131fb20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d29a20e5614b4436a6f63810e9d045e8" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 38425 } } peers { permanent_uuid: "93677f8a09994560884d95d0948cd44b" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 33659 } } peers { permanent_uuid: "067a1b0f46d248bb90f55d942131fb20" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 39187 } }
I20260806 02:58:34.639532 655 leader_election.cc:288] T d78670f612f344a18730c5c3c2ae38f3 P 067a1b0f46d248bb90f55d942131fb20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d29a20e5614b4436a6f63810e9d045e8 (127.31.74.194:38425), 93677f8a09994560884d95d0948cd44b (127.31.74.195:33659)
I20260806 02:58:34.649590 644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:34.640036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.193:57659 (local address 127.31.74.194:38425)
0806 02:58:34.640323 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:34.640333 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:34.641497 (+ 1164us) server_negotiation.cc:407] Connection header received
0806 02:58:34.641710 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:34.641716 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:34.641796 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:34.641873 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:58:34.642776 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:34.643761 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:34.646745 (+ 2984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:34.648063 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:34.648260 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:34.648339 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:34.649075 (+ 736us) server_negotiation.cc:299] Negotiation successful
0806 02:58:34.649239 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260806 02:58:34.649570 646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:34.640497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.193:37075 (local address 127.31.74.195:33659)
0806 02:58:34.640762 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:34.640772 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:34.641051 (+ 279us) server_negotiation.cc:407] Connection header received
0806 02:58:34.641340 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:34.641350 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:34.641418 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:34.641546 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:58:34.642252 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:34.643428 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:34.646736 (+ 3308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:34.648063 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:34.648261 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:34.648339 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:34.649075 (+ 736us) server_negotiation.cc:299] Negotiation successful
0806 02:58:34.649240 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260806 02:58:34.649842 658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:34.639995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.195:33659 (local address 127.31.74.193:37075)
0806 02:58:34.640910 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:34.640932 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 02:58:34.641146 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:58:34.641583 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:58:34.641592 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:58:34.641635 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:58:34.642029 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:34.642061 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:34.643620 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:34.643624 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:34.646487 (+ 2863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:34.646499 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:34.648420 (+ 1921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:34.648432 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:34.648839 (+ 407us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:34.648868 (+ 29us) client_negotiation.cc:769] Sending connection context
0806 02:58:34.648978 (+ 110us) client_negotiation.cc:240] Negotiation successful
0806 02:58:34.649289 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":779,"mutex_wait_us":198,"thread_start_us":565,"threads_started":1}
I20260806 02:58:34.649864 659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:34.640096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.194:38425 (local address 127.31.74.193:57659)
0806 02:58:34.641370 (+ 1274us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:34.641386 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:58:34.641542 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:58:34.641929 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:58:34.641934 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:58:34.641971 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:58:34.642503 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:34.642518 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:34.643913 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:34.643919 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:34.646478 (+ 2559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:34.646499 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:34.648462 (+ 1963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:34.648470 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:34.648774 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:34.648852 (+ 78us) client_negotiation.cc:769] Sending connection context
0806 02:58:34.648978 (+ 126us) client_negotiation.cc:240] Negotiation successful
0806 02:58:34.649289 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1137,"mutex_wait_us":140,"thread_start_us":907,"threads_started":1}
I20260806 02:58:34.651958 582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d78670f612f344a18730c5c3c2ae38f3" candidate_uuid: "067a1b0f46d248bb90f55d942131fb20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93677f8a09994560884d95d0948cd44b" is_pre_election: true
I20260806 02:58:34.651960 445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d78670f612f344a18730c5c3c2ae38f3" candidate_uuid: "067a1b0f46d248bb90f55d942131fb20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d29a20e5614b4436a6f63810e9d045e8" is_pre_election: true
I20260806 02:58:34.652717 582 raft_consensus.cc:2454] T d78670f612f344a18730c5c3c2ae38f3 P 93677f8a09994560884d95d0948cd44b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 067a1b0f46d248bb90f55d942131fb20 in term 0.
I20260806 02:58:34.653791 32712 leader_election.cc:302] T d78670f612f344a18730c5c3c2ae38f3 P 067a1b0f46d248bb90f55d942131fb20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 067a1b0f46d248bb90f55d942131fb20, 93677f8a09994560884d95d0948cd44b; no voters:
I20260806 02:58:34.654399 655 raft_consensus.cc:2781] T d78670f612f344a18730c5c3c2ae38f3 P 067a1b0f46d248bb90f55d942131fb20 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:58:34.654575 655 raft_consensus.cc:482] T d78670f612f344a18730c5c3c2ae38f3 P 067a1b0f46d248bb90f55d942131fb20 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:58:34.654740 655 raft_consensus.cc:3037] T d78670f612f344a18730c5c3c2ae38f3 P 067a1b0f46d248bb90f55d942131fb20 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:58:34.655865 445 raft_consensus.cc:2454] T d78670f612f344a18730c5c3c2ae38f3 P d29a20e5614b4436a6f63810e9d045e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 067a1b0f46d248bb90f55d942131fb20 in term 0.
I20260806 02:58:34.657624 655 raft_consensus.cc:504] T d78670f612f344a18730c5c3c2ae38f3 P 067a1b0f46d248bb90f55d942131fb20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d29a20e5614b4436a6f63810e9d045e8" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 38425 } } peers { permanent_uuid: "93677f8a09994560884d95d0948cd44b" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 33659 } } peers { permanent_uuid: "067a1b0f46d248bb90f55d942131fb20" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 39187 } }
I20260806 02:58:34.658383 655 leader_election.cc:288] T d78670f612f344a18730c5c3c2ae38f3 P 067a1b0f46d248bb90f55d942131fb20 [CANDIDATE]: Term 1 election: Requested vote from peers d29a20e5614b4436a6f63810e9d045e8 (127.31.74.194:38425), 93677f8a09994560884d95d0948cd44b (127.31.74.195:33659)
I20260806 02:58:34.659158 582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d78670f612f344a18730c5c3c2ae38f3" candidate_uuid: "067a1b0f46d248bb90f55d942131fb20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93677f8a09994560884d95d0948cd44b"
I20260806 02:58:34.659258 445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d78670f612f344a18730c5c3c2ae38f3" candidate_uuid: "067a1b0f46d248bb90f55d942131fb20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d29a20e5614b4436a6f63810e9d045e8"
I20260806 02:58:34.659461 582 raft_consensus.cc:3037] T d78670f612f344a18730c5c3c2ae38f3 P 93677f8a09994560884d95d0948cd44b [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:58:34.659528 445 raft_consensus.cc:3037] T d78670f612f344a18730c5c3c2ae38f3 P d29a20e5614b4436a6f63810e9d045e8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:58:34.662037 582 raft_consensus.cc:2454] T d78670f612f344a18730c5c3c2ae38f3 P 93677f8a09994560884d95d0948cd44b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 067a1b0f46d248bb90f55d942131fb20 in term 1.
I20260806 02:58:34.662050 445 raft_consensus.cc:2454] T d78670f612f344a18730c5c3c2ae38f3 P d29a20e5614b4436a6f63810e9d045e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 067a1b0f46d248bb90f55d942131fb20 in term 1.
I20260806 02:58:34.662731 32712 leader_election.cc:302] T d78670f612f344a18730c5c3c2ae38f3 P 067a1b0f46d248bb90f55d942131fb20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 067a1b0f46d248bb90f55d942131fb20, 93677f8a09994560884d95d0948cd44b; no voters:
I20260806 02:58:34.663120 655 raft_consensus.cc:2781] T d78670f612f344a18730c5c3c2ae38f3 P 067a1b0f46d248bb90f55d942131fb20 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:58:34.663709 655 raft_consensus.cc:686] T d78670f612f344a18730c5c3c2ae38f3 P 067a1b0f46d248bb90f55d942131fb20 [term 1 LEADER]: Becoming Leader. State: Replica: 067a1b0f46d248bb90f55d942131fb20, State: Running, Role: LEADER
I20260806 02:58:34.664352 655 consensus_queue.cc:237] T d78670f612f344a18730c5c3c2ae38f3 P 067a1b0f46d248bb90f55d942131fb20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d29a20e5614b4436a6f63810e9d045e8" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 38425 } } peers { permanent_uuid: "93677f8a09994560884d95d0948cd44b" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 33659 } } peers { permanent_uuid: "067a1b0f46d248bb90f55d942131fb20" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 39187 } }
I20260806 02:58:34.671659 32630 catalog_manager.cc:5495] T d78670f612f344a18730c5c3c2ae38f3 P 067a1b0f46d248bb90f55d942131fb20 reported cstate change: term changed from 0 to 1, leader changed from <none> to 067a1b0f46d248bb90f55d942131fb20 (127.31.74.193). New cstate: current_term: 1 leader_uuid: "067a1b0f46d248bb90f55d942131fb20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d29a20e5614b4436a6f63810e9d045e8" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 38425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93677f8a09994560884d95d0948cd44b" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 33659 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "067a1b0f46d248bb90f55d942131fb20" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 39187 } health_report { overall_health: HEALTHY } } }
Aug 06 02:58:34 dist-test-slave-dhph krb5kdc[32585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.74.193@KRBTEST.COM
I20260806 02:58:34.768852 648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:34.748161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57064 (local address 127.31.74.193:39187)
0806 02:58:34.748463 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:34.748474 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:34.749128 (+ 654us) server_negotiation.cc:407] Connection header received
0806 02:58:34.749271 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:34.749277 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:34.749354 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:34.749478 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:34.751381 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:34.752221 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:34.754086 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:34.754677 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:34.754816 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:34.760560 (+ 5744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:58:34.760580 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:58:34.760597 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:58:34.760647 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:58:34.764506 (+ 3859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:34.765452 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:34.765460 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:34.765462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:34.765618 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:34.766176 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:34.766182 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:34.766185 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:34.766977 (+ 792us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:58:34.767351 (+ 374us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:34.768149 (+ 798us) server_negotiation.cc:299] Negotiation successful
0806 02:58:34.768527 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
Aug 06 02:58:34 dist-test-slave-dhph krb5kdc[32585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985114, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 02:58:34 dist-test-slave-dhph krb5kdc[32585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985114, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.74.193@KRBTEST.COM
I20260806 02:58:34.843055 648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:34.822760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57068 (local address 127.31.74.193:39187)
0806 02:58:34.822978 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:34.822983 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:34.823607 (+ 624us) server_negotiation.cc:407] Connection header received
0806 02:58:34.823846 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:34.823860 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:34.823935 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:34.824058 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:34.825739 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:34.826657 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:34.828941 (+ 2284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:34.829628 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:34.829789 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:34.835638 (+ 5849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:58:34.835645 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:58:34.835649 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:58:34.835728 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:58:34.839257 (+ 3529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:34.840077 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:34.840084 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:34.840088 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:34.840244 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:34.840994 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:34.841002 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:34.841006 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:34.841361 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:58:34.841806 (+ 445us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:34.842077 (+ 271us) server_negotiation.cc:299] Negotiation successful
0806 02:58:34.842677 (+ 600us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
Aug 06 02:58:34 dist-test-slave-dhph krb5kdc[32585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985114, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 02:58:34 dist-test-slave-dhph krb5kdc[32585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985114, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.74.193@KRBTEST.COM
I20260806 02:58:34.912320 648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:34.890753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57078 (local address 127.31.74.193:39187)
0806 02:58:34.891016 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:34.891027 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:34.891678 (+ 651us) server_negotiation.cc:407] Connection header received
0806 02:58:34.891865 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:34.891870 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:34.891977 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:34.892076 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:34.893656 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:34.894523 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:34.896549 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:34.897327 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:34.897529 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:34.904135 (+ 6606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:58:34.904142 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:58:34.904146 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:58:34.904207 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:58:34.908295 (+ 4088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:34.909669 (+ 1374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:34.909675 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:34.909679 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:34.909857 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:34.910716 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:34.910720 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:34.910723 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:34.911056 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:58:34.911308 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:34.911722 (+ 414us) server_negotiation.cc:299] Negotiation successful
0806 02:58:34.911936 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260806 02:58:34.915024 32736 tablet_service.cc:2908] Rejecting scan request for tablet d78670f612f344a18730c5c3c2ae38f3: Uninitialized: clean time has not yet been initialized
I20260806 02:58:34.919965 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 32693
I20260806 02:58:34.941345 648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:34.921413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57082 (local address 127.31.74.193:39187)
0806 02:58:34.921625 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:34.921630 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:34.922650 (+ 1020us) server_negotiation.cc:407] Connection header received
0806 02:58:34.922900 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:34.922905 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:34.922981 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:34.923186 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:34.925887 (+ 2701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:34.927728 (+ 1841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:34.929795 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:34.930797 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:34.930931 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:34.933712 (+ 2781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:58:34.933724 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:58:34.933729 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:58:34.933862 (+ 133us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:58:34.938140 (+ 4278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:34.939082 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:34.939087 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:34.939089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:34.939252 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:34.939731 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:34.939736 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:34.939738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:34.940060 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:58:34.940289 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:34.940845 (+ 556us) server_negotiation.cc:299] Negotiation successful
0806 02:58:34.941039 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 02:58:34.942693 350 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:58:35.056589 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 32693
I20260806 02:58:35.078792 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 361
Aug 06 02:58:35 dist-test-slave-dhph krb5kdc[32585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985114, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.74.194@KRBTEST.COM
I20260806 02:58:35.101075 644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:35.079697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52120 (local address 127.31.74.194:38425)
0806 02:58:35.079874 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:35.079878 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:35.079897 (+ 19us) server_negotiation.cc:407] Connection header received
0806 02:58:35.080012 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:35.080016 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:35.080093 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:35.080173 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:35.082249 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:35.083329 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:35.084948 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:35.085620 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:35.086207 (+ 587us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:35.092113 (+ 5906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:58:35.092142 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:58:35.092166 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:58:35.092234 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:58:35.096663 (+ 4429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:35.097724 (+ 1061us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:35.097735 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:35.097738 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:35.097991 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:35.098528 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:35.098532 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:35.098535 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:35.099271 (+ 736us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:58:35.099630 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:35.100211 (+ 581us) server_negotiation.cc:299] Negotiation successful
0806 02:58:35.100543 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 02:58:35.102161 486 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:58:35.225047 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 361
I20260806 02:58:35.248857 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 497
Aug 06 02:58:35 dist-test-slave-dhph krb5kdc[32585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985114, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.74.195@KRBTEST.COM
I20260806 02:58:35.269013 675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:35.249857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35104 (local address 127.31.74.195:33659)
0806 02:58:35.250410 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:35.250415 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:35.250432 (+ 17us) server_negotiation.cc:407] Connection header received
0806 02:58:35.250489 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:35.250493 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:35.250556 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:35.250756 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:35.252126 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:35.252906 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:35.254478 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:35.255036 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:35.255144 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:35.260844 (+ 5700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:58:35.260865 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:58:35.260883 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:58:35.260930 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:58:35.265137 (+ 4207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:35.266470 (+ 1333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:35.266479 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:35.266482 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:35.266759 (+ 277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:35.267337 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:35.267341 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:35.267343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:35.267920 (+ 577us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:58:35.268322 (+ 402us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:35.268488 (+ 166us) server_negotiation.cc:299] Negotiation successful
0806 02:58:35.268691 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":294,"threads_started":1}
I20260806 02:58:35.271098 624 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:58:35.372473 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 497
W20260806 02:58:35.395941 32616 connection.cc:535] server connection from 127.31.74.195:52427 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260806 02:58:35.396129 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 32598
I20260806 02:58:35.397585 32661 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:58:35.527011 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 32598
2026-08-06T02:58:35Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (5354 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:58:35 dist-test-slave-dhph krb5kdc[680](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:58:35 dist-test-slave-dhph krb5kdc[680](info): set up 2 sockets
Aug 06 02:58:35 dist-test-slave-dhph krb5kdc[680](info): commencing operation
krb5kdc: starting...
W20260806 02:58:36.875021 32043 mini_kdc.cc:121] Time spent starting KDC: real 1.291s user 0.001s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/krb5kdc/test-user.keytab.
Aug 06 02:58:37 dist-test-slave-dhph krb5kdc[680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985117, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:58:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:58:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.74.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab.
Entry for principal kudu/127.31.74.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab.
I20260806 02:58:37.139482 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:37001
--webserver_interface=127.31.74.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab
--principal=kudu/127.31.74.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:33219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.74.254:37001
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:58:37.514433 693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:58:37.514781 693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:58:37.514874 693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:58:37.522294 693 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:58:37.522428 693 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260806 02:58:37.522469 693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:58:37.522524 693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:58:37.522635 693 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:58:37.522725 693 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:58:37.534734 693 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:33219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.74.254:37001
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab
--principal=kudu/127.31.74.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:37001
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.74.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:58:37.536073 693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:58:37.537374 693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:58:37.550434 699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:58:37.550570 701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:58:37.550629 698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:58:37.551437 693 server_base.cc:1034] running on GCE node
I20260806 02:58:37.553180 693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:58:37.556106 693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:58:37.557480 693 hybrid_clock.cc:648] HybridClock initialized: now 1785985117557418 us; error 70 us; skew 500 ppm
I20260806 02:58:37.557914 693 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:58:37 dist-test-slave-dhph krb5kdc[680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985117, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:58:37.561838 693 init.cc:377] Logged in from keytab as kudu/127.31.74.254@KRBTEST.COM (short username kudu)
I20260806 02:58:37.563709 693 webserver.cc:460] Webserver started at http://127.31.74.254:44769/ using document root <none> and password file <none>
I20260806 02:58:37.564505 693 fs_manager.cc:359] Metadata directory not provided
I20260806 02:58:37.564652 693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:58:37.565012 693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:58:37.568325 693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/minicluster-data/master-0/data/instance:
uuid: "b5e6b82242fc46519f1bfc172a68c982"
format_stamp: "Formatted at 2026-08-06 02:58:37 on dist-test-slave-dhph"
server_key: "3db6f65df3025465a766c6e56f84fc85"
server_key_iv: "a95d8b4294b893098ea7433f1862427d"
server_key_version: "encryptionkey@0"
I20260806 02:58:37.569171 693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/minicluster-data/master-0/wal/instance:
uuid: "b5e6b82242fc46519f1bfc172a68c982"
format_stamp: "Formatted at 2026-08-06 02:58:37 on dist-test-slave-dhph"
server_key: "3db6f65df3025465a766c6e56f84fc85"
server_key_iv: "a95d8b4294b893098ea7433f1862427d"
server_key_version: "encryptionkey@0"
I20260806 02:58:37.575201 693 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260806 02:58:37.579967 708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:58:37.582096 693 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:58:37.582334 693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/minicluster-data/master-0/wal
uuid: "b5e6b82242fc46519f1bfc172a68c982"
format_stamp: "Formatted at 2026-08-06 02:58:37 on dist-test-slave-dhph"
server_key: "3db6f65df3025465a766c6e56f84fc85"
server_key_iv: "a95d8b4294b893098ea7433f1862427d"
server_key_version: "encryptionkey@0"
I20260806 02:58:37.582588 693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:58:37.600198 693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:58:37.610507 693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:58:37.611186 693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:58:37.639313 693 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.254:37001
I20260806 02:58:37.639353 772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.254:37001 every 8 connection(s)
I20260806 02:58:37.641139 693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
I20260806 02:58:37.646098 773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:58:37.646488 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 693
I20260806 02:58:37.646776 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/minicluster-data/master-0/wal/instance
I20260806 02:58:37.647441 32043 external_mini_cluster.cc:1428] Setting key 179cdc77d9287e4f8d4ceccf45aed6af
I20260806 02:58:37.666772 773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5e6b82242fc46519f1bfc172a68c982: Bootstrap starting.
Aug 06 02:58:37 dist-test-slave-dhph krb5kdc[680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985117, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 02:58:37.671892 773 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5e6b82242fc46519f1bfc172a68c982: Neither blocks nor log segments found. Creating new log.
I20260806 02:58:37.673456 773 log.cc:824] T 00000000000000000000000000000000 P b5e6b82242fc46519f1bfc172a68c982: Log is configured to *not* fsync() on all Append() calls
I20260806 02:58:37.677575 773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5e6b82242fc46519f1bfc172a68c982: No bootstrap required, opened a new log
I20260806 02:58:37.681459 776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:37.649596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39800 (local address 127.31.74.254:37001)
0806 02:58:37.650572 (+ 976us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:37.650641 (+ 69us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:37.650702 (+ 61us) server_negotiation.cc:407] Connection header received
0806 02:58:37.654141 (+ 3439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:37.654176 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:37.654779 (+ 603us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:37.655172 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:37.658006 (+ 2834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:37.659538 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:37.661937 (+ 2399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:37.663192 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:37.663380 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:37.670838 (+ 7458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:58:37.670853 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:58:37.670875 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:58:37.670935 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:58:37.675915 (+ 4980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:37.676793 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:37.676801 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:37.676810 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:37.677057 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:37.677464 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:37.677470 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:37.677474 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:37.678284 (+ 810us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:58:37.678632 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:37.679172 (+ 540us) server_negotiation.cc:299] Negotiation successful
0806 02:58:37.679659 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":698,"thread_start_us":476,"threads_started":1}
I20260806 02:58:37.688470 773 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5e6b82242fc46519f1bfc172a68c982 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5e6b82242fc46519f1bfc172a68c982" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 37001 } }
I20260806 02:58:37.688910 773 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5e6b82242fc46519f1bfc172a68c982 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:58:37.688987 773 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5e6b82242fc46519f1bfc172a68c982 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5e6b82242fc46519f1bfc172a68c982, State: Initialized, Role: FOLLOWER
I20260806 02:58:37.689553 773 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5e6b82242fc46519f1bfc172a68c982 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5e6b82242fc46519f1bfc172a68c982" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 37001 } }
I20260806 02:58:37.689790 773 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5e6b82242fc46519f1bfc172a68c982 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:58:37.689976 773 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5e6b82242fc46519f1bfc172a68c982 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:58:37.690181 773 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5e6b82242fc46519f1bfc172a68c982 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:58:37.693008 773 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5e6b82242fc46519f1bfc172a68c982 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5e6b82242fc46519f1bfc172a68c982" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 37001 } }
I20260806 02:58:37.693650 773 leader_election.cc:302] T 00000000000000000000000000000000 P b5e6b82242fc46519f1bfc172a68c982 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5e6b82242fc46519f1bfc172a68c982; no voters:
I20260806 02:58:37.694516 773 leader_election.cc:288] T 00000000000000000000000000000000 P b5e6b82242fc46519f1bfc172a68c982 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:58:37.694746 778 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5e6b82242fc46519f1bfc172a68c982 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:58:37.695945 778 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5e6b82242fc46519f1bfc172a68c982 [term 1 LEADER]: Becoming Leader. State: Replica: b5e6b82242fc46519f1bfc172a68c982, State: Running, Role: LEADER
I20260806 02:58:37.696979 773 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5e6b82242fc46519f1bfc172a68c982 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:58:37.696631 778 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5e6b82242fc46519f1bfc172a68c982 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5e6b82242fc46519f1bfc172a68c982" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 37001 } }
I20260806 02:58:37.702857 780 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5e6b82242fc46519f1bfc172a68c982 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5e6b82242fc46519f1bfc172a68c982. Latest consensus state: current_term: 1 leader_uuid: "b5e6b82242fc46519f1bfc172a68c982" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5e6b82242fc46519f1bfc172a68c982" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 37001 } } }
I20260806 02:58:37.703418 780 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5e6b82242fc46519f1bfc172a68c982 [sys.catalog]: This master's current role is: LEADER
I20260806 02:58:37.702885 779 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5e6b82242fc46519f1bfc172a68c982 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5e6b82242fc46519f1bfc172a68c982" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5e6b82242fc46519f1bfc172a68c982" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 37001 } } }
I20260806 02:58:37.705029 779 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5e6b82242fc46519f1bfc172a68c982 [sys.catalog]: This master's current role is: LEADER
I20260806 02:58:37.711424 791 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:58:37.719270 791 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:58:37.730710 791 catalog_manager.cc:1347] Generated new cluster ID: 7d6089cfc2c5440aa163ea9dc1a3774f
I20260806 02:58:37.730953 791 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:58:37.754516 791 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:58:37.756808 791 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:58:37.780746 791 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b5e6b82242fc46519f1bfc172a68c982: Generated new TSK 0
I20260806 02:58:37.781909 791 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.74.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.193.keytab.
Entry for principal kudu/127.31.74.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.193.keytab.
I20260806 02:58:37.858335 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.193:0
--local_ip_for_outbound_sockets=127.31.74.193
--webserver_interface=127.31.74.193
--webserver_port=0
--tserver_master_addrs=127.31.74.254:37001
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.193.keytab
--principal=kudu/127.31.74.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:33219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:58:38.256435 799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:58:38.256798 799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:58:38.256975 799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:58:38.265647 799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:58:38.265857 799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:58:38.266376 799 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 02:58:38.266537 799 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.193
I20260806 02:58:38.278877 799 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:33219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.193.keytab
--principal=kudu/127.31.74.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.74.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.74.254:37001
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.193
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:58:38.280453 799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:58:38.281877 799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:58:38.296535 807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:58:38.296778 805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:58:38.296515 804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:58:38.300817 799 server_base.cc:1034] running on GCE node
I20260806 02:58:38.301613 799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:58:38.302834 799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:58:38.304092 799 hybrid_clock.cc:648] HybridClock initialized: now 1785985118304010 us; error 54 us; skew 500 ppm
I20260806 02:58:38.304548 799 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:58:38 dist-test-slave-dhph krb5kdc[680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985118, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:58:38.308108 799 init.cc:377] Logged in from keytab as kudu/127.31.74.193@KRBTEST.COM (short username kudu)
I20260806 02:58:38.309908 799 webserver.cc:460] Webserver started at http://127.31.74.193:39269/ using document root <none> and password file <none>
I20260806 02:58:38.310768 799 fs_manager.cc:359] Metadata directory not provided
I20260806 02:58:38.310877 799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:58:38.311270 799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:58:38.314469 799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/minicluster-data/ts-0/data/instance:
uuid: "c5b5c4a10bc14bf6b7efe6cbdcfad492"
format_stamp: "Formatted at 2026-08-06 02:58:38 on dist-test-slave-dhph"
server_key: "291962b460755506c8b9269206be9bf4"
server_key_iv: "105b70395ce11befe9fa656de4664135"
server_key_version: "encryptionkey@0"
I20260806 02:58:38.315405 799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/minicluster-data/ts-0/wal/instance:
uuid: "c5b5c4a10bc14bf6b7efe6cbdcfad492"
format_stamp: "Formatted at 2026-08-06 02:58:38 on dist-test-slave-dhph"
server_key: "291962b460755506c8b9269206be9bf4"
server_key_iv: "105b70395ce11befe9fa656de4664135"
server_key_version: "encryptionkey@0"
I20260806 02:58:38.321421 799 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.003s
I20260806 02:58:38.326161 814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:58:38.328059 799 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 02:58:38.328249 799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/minicluster-data/ts-0/wal
uuid: "c5b5c4a10bc14bf6b7efe6cbdcfad492"
format_stamp: "Formatted at 2026-08-06 02:58:38 on dist-test-slave-dhph"
server_key: "291962b460755506c8b9269206be9bf4"
server_key_iv: "105b70395ce11befe9fa656de4664135"
server_key_version: "encryptionkey@0"
I20260806 02:58:38.328483 799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:58:38.355341 799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:58:38.361657 799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:58:38.362000 799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:58:38.365265 799 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:58:38.365394 799 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:58:38.365541 799 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:58:38.365636 799 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 02:58:38 dist-test-slave-dhph krb5kdc[680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985118, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.193@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 02:58:38.394201 824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:38.366171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.193:37103 (local address 127.31.74.254:37001)
0806 02:58:38.366722 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:38.366730 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:38.368165 (+ 1435us) server_negotiation.cc:407] Connection header received
0806 02:58:38.371231 (+ 3066us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:38.371237 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:38.371341 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:38.371476 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:38.374805 (+ 3329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:38.375969 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:38.378124 (+ 2155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:38.378937 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:38.379125 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:38.385674 (+ 6549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:58:38.385684 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:58:38.385691 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:58:38.385749 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:58:38.389598 (+ 3849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:38.391214 (+ 1616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:38.391220 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:38.391224 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:38.391380 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:38.391973 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:38.391981 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:38.391985 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:38.392361 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:58:38.392523 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:38.393568 (+ 1045us) server_negotiation.cc:299] Negotiation successful
0806 02:58:38.393816 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":270,"threads_started":1}
I20260806 02:58:38.395968 825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:38.366957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:37001 (local address 127.31.74.193:37103)
0806 02:58:38.367959 (+ 1002us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:38.368013 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 02:58:38.370824 (+ 2811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:58:38.371939 (+ 1115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:58:38.371956 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:58:38.373416 (+ 1460us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:58:38.374408 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:38.374445 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:38.376156 (+ 1711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:38.376161 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:38.377716 (+ 1555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:38.377731 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:38.379434 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:38.379446 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:38.380298 (+ 852us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:38.381397 (+ 1099us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:58:38.381457 (+ 60us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:58:38.385366 (+ 3909us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:58:38.389922 (+ 4556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:58:38.389944 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:58:38.389964 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:58:38.391059 (+ 1095us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:58:38.391506 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:58:38.391514 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:58:38.391518 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:58:38.391766 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:58:38.392508 (+ 742us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:58:38.392517 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:58:38.393098 (+ 581us) client_negotiation.cc:769] Sending connection context
0806 02:58:38.393465 (+ 367us) client_negotiation.cc:240] Negotiation successful
0806 02:58:38.394113 (+ 648us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":667,"thread_start_us":429,"threads_started":1}
I20260806 02:58:38.418905 799 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.193:36295
I20260806 02:58:38.418931 942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.193:36295 every 8 connection(s)
I20260806 02:58:38.420332 799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
I20260806 02:58:38.422904 943 heartbeater.cc:347] Connected to a master server at 127.31.74.254:37001
I20260806 02:58:38.423259 943 heartbeater.cc:464] Registering TS with master...
I20260806 02:58:38.423571 824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:38.405487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.193:34829 (local address 127.31.74.254:37001)
0806 02:58:38.405741 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:38.405755 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:38.406440 (+ 685us) server_negotiation.cc:407] Connection header received
0806 02:58:38.407044 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:38.407050 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:38.407149 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:38.407571 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:38.409084 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:38.409971 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:38.412913 (+ 2942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:38.413608 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:38.413778 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:38.416882 (+ 3104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:58:38.416887 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:58:38.416890 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:58:38.416928 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:58:38.420032 (+ 3104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:38.420855 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:38.420859 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:38.420861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:38.420993 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:38.421520 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:38.421525 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:38.421528 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:38.421984 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:58:38.422132 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:38.422906 (+ 774us) server_negotiation.cc:299] Negotiation successful
0806 02:58:38.423183 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 02:58:38.423838 914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:38.405571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:37001 (local address 127.31.74.193:34829)
0806 02:58:38.406260 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:38.406287 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0806 02:58:38.406533 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:58:38.407596 (+ 1063us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:58:38.407601 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:58:38.408245 (+ 644us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:58:38.408908 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:38.408926 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:38.410171 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:38.410174 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:38.412619 (+ 2445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:38.412648 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:38.414076 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:38.414081 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:38.414252 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:38.415167 (+ 915us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:58:38.415200 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:58:38.416560 (+ 1360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:58:38.420195 (+ 3635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:58:38.420201 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:58:38.420205 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:58:38.420705 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:58:38.421118 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:58:38.421125 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:58:38.421131 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:58:38.421347 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:58:38.422215 (+ 868us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:58:38.422225 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:58:38.422506 (+ 281us) client_negotiation.cc:769] Sending connection context
0806 02:58:38.422772 (+ 266us) client_negotiation.cc:240] Negotiation successful
0806 02:58:38.423177 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":366,"threads_started":1}
I20260806 02:58:38.424572 943 heartbeater.cc:511] Master 127.31.74.254:37001 requested a full tablet report, sending...
I20260806 02:58:38.427003 726 ts_manager.cc:194] Registered new tserver with Master: c5b5c4a10bc14bf6b7efe6cbdcfad492 (127.31.74.193:36295)
I20260806 02:58:38.428969 726 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.74.193@KRBTEST.COM'} at 127.31.74.193:37103
I20260806 02:58:38.429514 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 799
I20260806 02:58:38.429783 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/minicluster-data/ts-0/wal/instance
I20260806 02:58:38.430572 32043 external_mini_cluster.cc:1428] Setting key 0333489e4a5f7f2ce2930cb82c94b1de
WARNING: no policy specified for kudu/127.31.74.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.194.keytab.
Entry for principal kudu/127.31.74.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.194.keytab.
I20260806 02:58:38.487775 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.194:0
--local_ip_for_outbound_sockets=127.31.74.194
--webserver_interface=127.31.74.194
--webserver_port=0
--tserver_master_addrs=127.31.74.254:37001
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.194.keytab
--principal=kudu/127.31.74.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:33219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:58:38.900866 947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:58:38.901319 947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:58:38.901480 947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:58:38.911393 947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:58:38.911937 947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:58:38.912094 947 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 02:58:38.912307 947 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.194
I20260806 02:58:38.928483 947 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:33219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.194.keytab
--principal=kudu/127.31.74.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.74.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.74.254:37001
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.194
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:58:38.930195 947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:58:38.932643 947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:58:38.949862 953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:58:38.949888 952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:58:38.949954 955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:58:38.956984 947 server_base.cc:1034] running on GCE node
I20260806 02:58:38.958096 947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:58:38.959710 947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:58:38.961213 947 hybrid_clock.cc:648] HybridClock initialized: now 1785985118961102 us; error 71 us; skew 500 ppm
I20260806 02:58:38.961607 947 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:58:38 dist-test-slave-dhph krb5kdc[680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985118, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:58:38.966310 947 init.cc:377] Logged in from keytab as kudu/127.31.74.194@KRBTEST.COM (short username kudu)
I20260806 02:58:38.968325 947 webserver.cc:460] Webserver started at http://127.31.74.194:37619/ using document root <none> and password file <none>
I20260806 02:58:38.969213 947 fs_manager.cc:359] Metadata directory not provided
I20260806 02:58:38.969322 947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:58:38.969687 947 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:58:38.973615 947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/minicluster-data/ts-1/data/instance:
uuid: "66dd377b30ea48b8a33c9892e715e9f0"
format_stamp: "Formatted at 2026-08-06 02:58:38 on dist-test-slave-dhph"
server_key: "9a6845ace73e13ed375e6b98821e5c31"
server_key_iv: "030ea743f44886148fecf783b1118ea2"
server_key_version: "encryptionkey@0"
I20260806 02:58:38.974689 947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/minicluster-data/ts-1/wal/instance:
uuid: "66dd377b30ea48b8a33c9892e715e9f0"
format_stamp: "Formatted at 2026-08-06 02:58:38 on dist-test-slave-dhph"
server_key: "9a6845ace73e13ed375e6b98821e5c31"
server_key_iv: "030ea743f44886148fecf783b1118ea2"
server_key_version: "encryptionkey@0"
I20260806 02:58:38.981343 947 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260806 02:58:38.986505 962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:58:38.988404 947 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260806 02:58:38.988612 947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/minicluster-data/ts-1/wal
uuid: "66dd377b30ea48b8a33c9892e715e9f0"
format_stamp: "Formatted at 2026-08-06 02:58:38 on dist-test-slave-dhph"
server_key: "9a6845ace73e13ed375e6b98821e5c31"
server_key_iv: "030ea743f44886148fecf783b1118ea2"
server_key_version: "encryptionkey@0"
I20260806 02:58:38.988883 947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:58:39.012115 947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:58:39.017535 947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:58:39.018056 947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:58:39.022442 947 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:58:39.022629 947 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:58:39.022810 947 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:58:39.022910 947 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 02:58:39 dist-test-slave-dhph krb5kdc[680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985118, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.194@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 02:58:39.059065 971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:39.023298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.194:46535 (local address 127.31.74.254:37001)
0806 02:58:39.024128 (+ 830us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:39.024133 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:39.025566 (+ 1433us) server_negotiation.cc:407] Connection header received
0806 02:58:39.028572 (+ 3006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:39.028578 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:39.028681 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:39.028807 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:39.031533 (+ 2726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:39.032904 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:39.038412 (+ 5508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:39.039075 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:39.039262 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:39.046365 (+ 7103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:58:39.046373 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:58:39.046377 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:58:39.046421 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:58:39.050906 (+ 4485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:39.052886 (+ 1980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:39.052899 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:39.052906 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:39.053131 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:39.054195 (+ 1064us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:39.054202 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:39.054207 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:39.054769 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:58:39.055000 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:39.056919 (+ 1919us) server_negotiation.cc:299] Negotiation successful
0806 02:58:39.057380 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":707,"thread_start_us":592,"threads_started":1}
I20260806 02:58:39.060595 972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:39.024239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:37001 (local address 127.31.74.194:46535)
0806 02:58:39.025363 (+ 1124us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:39.025416 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 02:58:39.027920 (+ 2504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:58:39.029097 (+ 1177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:58:39.029106 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:58:39.029948 (+ 842us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:58:39.031220 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:39.031263 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:39.033330 (+ 2067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:39.033338 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:39.038147 (+ 4809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:39.038172 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:39.039451 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:39.039460 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:39.039889 (+ 429us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:39.041076 (+ 1187us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:58:39.041126 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:58:39.045891 (+ 4765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:58:39.051123 (+ 5232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:58:39.051133 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:58:39.051151 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:58:39.052644 (+ 1493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:58:39.053357 (+ 713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:58:39.053367 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:58:39.053371 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:58:39.053844 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:58:39.055120 (+ 1276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:58:39.055130 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:58:39.056058 (+ 928us) client_negotiation.cc:769] Sending connection context
0806 02:58:39.056608 (+ 550us) client_negotiation.cc:240] Negotiation successful
0806 02:58:39.057589 (+ 981us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":744,"thread_start_us":411,"threads_started":1}
I20260806 02:58:39.086227 947 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.194:38711
I20260806 02:58:39.086285 1085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.194:38711 every 8 connection(s)
I20260806 02:58:39.087978 947 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
I20260806 02:58:39.090797 1086 heartbeater.cc:347] Connected to a master server at 127.31.74.254:37001
I20260806 02:58:39.091619 971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:39.068996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.194:51609 (local address 127.31.74.254:37001)
0806 02:58:39.069289 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:39.069299 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:39.070531 (+ 1232us) server_negotiation.cc:407] Connection header received
0806 02:58:39.070776 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:39.070786 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:39.070902 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:39.071039 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:39.073774 (+ 2735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:39.075011 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:39.077853 (+ 2842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:39.078501 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:39.078705 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:39.082950 (+ 4245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:58:39.082963 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:58:39.082968 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:58:39.083007 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:58:39.087374 (+ 4367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:39.088298 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:39.088302 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:39.088305 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:39.088483 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:39.089300 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:39.089305 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:39.089307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:39.089914 (+ 607us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:58:39.090128 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:39.090834 (+ 706us) server_negotiation.cc:299] Negotiation successful
0806 02:58:39.091098 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260806 02:58:39.092490 1086 heartbeater.cc:464] Registering TS with master...
I20260806 02:58:39.092547 1051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:39.068991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:37001 (local address 127.31.74.194:51609)
0806 02:58:39.070383 (+ 1392us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:39.070407 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 02:58:39.070570 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:58:39.071327 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:58:39.071335 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:58:39.072520 (+ 1185us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:58:39.073464 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:39.073487 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:39.075205 (+ 1718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:39.075213 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:39.077619 (+ 2406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:39.077646 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:39.078835 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:39.078845 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:39.079095 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:39.080490 (+ 1395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:58:39.080522 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:58:39.082502 (+ 1980us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:58:39.087546 (+ 5044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:58:39.087552 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:58:39.087562 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:58:39.088138 (+ 576us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:58:39.088655 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:58:39.088660 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:58:39.088663 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:58:39.088862 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:58:39.090079 (+ 1217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:58:39.090085 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:58:39.090331 (+ 246us) client_negotiation.cc:769] Sending connection context
0806 02:58:39.090519 (+ 188us) client_negotiation.cc:240] Negotiation successful
0806 02:58:39.091784 (+ 1265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1217,"thread_start_us":1006,"threads_started":1}
I20260806 02:58:39.094292 1086 heartbeater.cc:511] Master 127.31.74.254:37001 requested a full tablet report, sending...
I20260806 02:58:39.096673 726 ts_manager.cc:194] Registered new tserver with Master: 66dd377b30ea48b8a33c9892e715e9f0 (127.31.74.194:38711)
I20260806 02:58:39.097688 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 947
I20260806 02:58:39.097916 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/minicluster-data/ts-1/wal/instance
I20260806 02:58:39.097884 726 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.74.194@KRBTEST.COM'} at 127.31.74.194:46535
I20260806 02:58:39.098541 32043 external_mini_cluster.cc:1428] Setting key b0426f86cd1439c71d7441b2a834761b
WARNING: no policy specified for kudu/127.31.74.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.195.keytab.
Entry for principal kudu/127.31.74.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.195.keytab.
I20260806 02:58:39.169835 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.195:0
--local_ip_for_outbound_sockets=127.31.74.195
--webserver_interface=127.31.74.195
--webserver_port=0
--tserver_master_addrs=127.31.74.254:37001
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.195.keytab
--principal=kudu/127.31.74.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:33219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:58:39.433350 943 heartbeater.cc:503] Master 127.31.74.254:37001 was elected leader, sending a full tablet report...
W20260806 02:58:39.614655 1090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:58:39.615011 1090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:58:39.615175 1090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:58:39.624595 1090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:58:39.624770 1090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:58:39.624946 1090 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 02:58:39.625080 1090 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.195
I20260806 02:58:39.638659 1090 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:33219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.195.keytab
--principal=kudu/127.31.74.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.74.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.74.254:37001
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.195
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:58:39.641201 1090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:58:39.642930 1090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:58:39.661078 1099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:58:39.660998 1096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:58:39.661029 1097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:58:39.662338 1090 server_base.cc:1034] running on GCE node
I20260806 02:58:39.663415 1090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:58:39.664870 1090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:58:39.666148 1090 hybrid_clock.cc:648] HybridClock initialized: now 1785985119666082 us; error 35 us; skew 500 ppm
I20260806 02:58:39.666555 1090 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:58:39 dist-test-slave-dhph krb5kdc[680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985119, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:58:39.670269 1090 init.cc:377] Logged in from keytab as kudu/127.31.74.195@KRBTEST.COM (short username kudu)
I20260806 02:58:39.672374 1090 webserver.cc:460] Webserver started at http://127.31.74.195:40901/ using document root <none> and password file <none>
I20260806 02:58:39.673205 1090 fs_manager.cc:359] Metadata directory not provided
I20260806 02:58:39.673314 1090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:58:39.673692 1090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:58:39.677315 1090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/minicluster-data/ts-2/data/instance:
uuid: "f9b291a1bfb24cb0af222df109aac8ce"
format_stamp: "Formatted at 2026-08-06 02:58:39 on dist-test-slave-dhph"
server_key: "77ed9b5a7cc39ad89bbe669a1206a0a7"
server_key_iv: "83dede95ab0751c54269496ab917bd78"
server_key_version: "encryptionkey@0"
I20260806 02:58:39.678226 1090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/minicluster-data/ts-2/wal/instance:
uuid: "f9b291a1bfb24cb0af222df109aac8ce"
format_stamp: "Formatted at 2026-08-06 02:58:39 on dist-test-slave-dhph"
server_key: "77ed9b5a7cc39ad89bbe669a1206a0a7"
server_key_iv: "83dede95ab0751c54269496ab917bd78"
server_key_version: "encryptionkey@0"
I20260806 02:58:39.685560 1090 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.003s
I20260806 02:58:39.690110 1106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:58:39.691946 1090 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260806 02:58:39.692155 1090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/minicluster-data/ts-2/wal
uuid: "f9b291a1bfb24cb0af222df109aac8ce"
format_stamp: "Formatted at 2026-08-06 02:58:39 on dist-test-slave-dhph"
server_key: "77ed9b5a7cc39ad89bbe669a1206a0a7"
server_key_iv: "83dede95ab0751c54269496ab917bd78"
server_key_version: "encryptionkey@0"
I20260806 02:58:39.692422 1090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:58:39.710465 1090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:58:39.715947 1090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:58:39.716270 1090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:58:39.719897 1090 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:58:39.720012 1090 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:58:39.720147 1090 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:58:39.720248 1090 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 02:58:39 dist-test-slave-dhph krb5kdc[680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985119, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.195@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 02:58:39.752252 1116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:39.720509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.195:43103 (local address 127.31.74.254:37001)
0806 02:58:39.721402 (+ 893us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:39.721407 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:39.722511 (+ 1104us) server_negotiation.cc:407] Connection header received
0806 02:58:39.724952 (+ 2441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:39.724958 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:39.725066 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:39.725199 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:39.728112 (+ 2913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:39.729347 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:39.734139 (+ 4792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:39.734875 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:39.735067 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:39.741581 (+ 6514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:58:39.741592 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:58:39.741600 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:58:39.741664 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:58:39.746014 (+ 4350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:39.747874 (+ 1860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:39.747884 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:39.747890 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:39.748122 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:39.748922 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:39.748929 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:39.748932 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:39.749374 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:58:39.749609 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:39.751250 (+ 1641us) server_negotiation.cc:299] Negotiation successful
0806 02:58:39.751695 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":767,"thread_start_us":650,"threads_started":1}
I20260806 02:58:39.754161 1117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:39.721309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:37001 (local address 127.31.74.195:43103)
0806 02:58:39.722344 (+ 1035us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:39.722389 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 02:58:39.724612 (+ 2223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:58:39.725553 (+ 941us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:58:39.725571 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:58:39.726709 (+ 1138us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:58:39.727864 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:39.727898 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:39.729590 (+ 1692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:39.729597 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:39.733686 (+ 4089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:39.733737 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:39.735350 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:39.735369 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:39.735807 (+ 438us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:39.736830 (+ 1023us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:58:39.736877 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:58:39.741156 (+ 4279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:58:39.746244 (+ 5088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:58:39.746259 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:58:39.746309 (+ 50us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:58:39.747613 (+ 1304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:58:39.748316 (+ 703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:58:39.748325 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:58:39.748331 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:58:39.748726 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:58:39.749601 (+ 875us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:58:39.749611 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:58:39.750289 (+ 678us) client_negotiation.cc:769] Sending connection context
0806 02:58:39.750750 (+ 461us) client_negotiation.cc:240] Negotiation successful
0806 02:58:39.751693 (+ 943us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":407,"threads_started":1}
I20260806 02:58:39.774642 1090 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.195:37639
I20260806 02:58:39.774665 1229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.195:37639 every 8 connection(s)
I20260806 02:58:39.776638 1090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
I20260806 02:58:39.779610 1230 heartbeater.cc:347] Connected to a master server at 127.31.74.254:37001
I20260806 02:58:39.779909 1230 heartbeater.cc:464] Registering TS with master...
I20260806 02:58:39.780813 1230 heartbeater.cc:511] Master 127.31.74.254:37001 requested a full tablet report, sending...
I20260806 02:58:39.781001 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 1090
I20260806 02:58:39.781234 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/minicluster-data/ts-2/wal/instance
I20260806 02:58:39.781920 32043 external_mini_cluster.cc:1428] Setting key 5dc7b17056e9b0f2b1944cb0382c8a8d
I20260806 02:58:39.782919 726 ts_manager.cc:194] Registered new tserver with Master: f9b291a1bfb24cb0af222df109aac8ce (127.31.74.195:37639)
I20260806 02:58:39.784253 726 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.74.195@KRBTEST.COM'} at 127.31.74.195:43103
I20260806 02:58:39.785841 32043 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:58:39.787765 1222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:39.770172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:37001 (local address 127.31.74.195:48433)
0806 02:58:39.771109 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:39.771126 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:58:39.771310 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:58:39.771752 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:58:39.771762 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:58:39.772590 (+ 828us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:58:39.773055 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:39.773072 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:39.774826 (+ 1754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:39.774834 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:39.776552 (+ 1718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:39.776572 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:39.777970 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:39.777978 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:39.778318 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:39.779404 (+ 1086us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:58:39.779438 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:58:39.780479 (+ 1041us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:58:39.784723 (+ 4244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:58:39.784731 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:58:39.784735 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:58:39.785267 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:58:39.785736 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:58:39.785741 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:58:39.785746 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:58:39.785931 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:58:39.786712 (+ 781us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:58:39.786717 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:58:39.786917 (+ 200us) client_negotiation.cc:769] Sending connection context
0806 02:58:39.787074 (+ 157us) client_negotiation.cc:240] Negotiation successful
0806 02:58:39.787368 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":779,"mutex_wait_us":1,"thread_start_us":485,"threads_started":1}
I20260806 02:58:39.788686 1116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:39.770154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.195:48433 (local address 127.31.74.254:37001)
0806 02:58:39.770415 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:39.770423 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:39.771224 (+ 801us) server_negotiation.cc:407] Connection header received
0806 02:58:39.771447 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:39.771453 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:39.771555 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:39.771673 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:39.773337 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:39.774638 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:39.776750 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:39.777474 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:39.777654 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:39.780721 (+ 3067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:58:39.780729 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:58:39.780734 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:58:39.780777 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:58:39.784517 (+ 3740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:39.785405 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:39.785411 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:39.785414 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:39.785591 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:39.786045 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:39.786052 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:39.786056 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:39.786528 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:58:39.786773 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:39.788250 (+ 1477us) server_negotiation.cc:299] Negotiation successful
0806 02:58:39.788442 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
Aug 06 02:58:39 dist-test-slave-dhph krb5kdc[680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985119, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 02:58:39 dist-test-slave-dhph krb5kdc[680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985119, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 02:58:39.850392 1116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:39.833682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39804 (local address 127.31.74.254:37001)
0806 02:58:39.833980 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:39.833991 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:39.834417 (+ 426us) server_negotiation.cc:407] Connection header received
0806 02:58:39.834701 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:39.834706 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:39.834779 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:39.834894 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:39.837055 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:39.837841 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:39.839308 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:39.839736 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:39.839879 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:39.844267 (+ 4388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:58:39.844272 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:58:39.844281 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:58:39.844325 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:58:39.847535 (+ 3210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:39.848306 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:39.848310 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:39.848312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:39.848452 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:39.848855 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:39.848860 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:39.848863 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:39.849159 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:58:39.849504 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:39.849775 (+ 271us) server_negotiation.cc:299] Negotiation successful
0806 02:58:39.850016 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260806 02:58:39.855890 725 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39804:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:58:39.863843 725 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:58:39.896751 1243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:39.880410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:36295 (local address 127.0.0.1:49994)
0806 02:58:39.883625 (+ 3215us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:39.883684 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0806 02:58:39.883951 (+ 267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:58:39.885909 (+ 1958us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:58:39.885917 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:58:39.885958 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:58:39.886549 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:39.886561 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:39.889276 (+ 2715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:39.889284 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:39.892650 (+ 3366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:39.892670 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:39.894696 (+ 2026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:39.894706 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:39.895700 (+ 994us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:39.895766 (+ 66us) client_negotiation.cc:769] Sending connection context
0806 02:58:39.895908 (+ 142us) client_negotiation.cc:240] Negotiation successful
0806 02:58:39.896124 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2755,"mutex_wait_us":654,"thread_start_us":2880,"threads_started":1}
I20260806 02:58:39.898027 1244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:39.880826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49994 (local address 127.31.74.193:36295)
0806 02:58:39.885222 (+ 4396us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:39.885228 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:39.885253 (+ 25us) server_negotiation.cc:407] Connection header received
0806 02:58:39.885343 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:39.885350 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:39.885628 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:39.886819 (+ 1191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:58:39.887020 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:39.889050 (+ 2030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:39.892969 (+ 3919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:39.894387 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:39.894799 (+ 412us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:39.895073 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:39.897123 (+ 2050us) server_negotiation.cc:299] Negotiation successful
0806 02:58:39.897462 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4232,"thread_start_us":4077,"threads_started":1}
I20260806 02:58:39.899556 1246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:39.881883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40830 (local address 127.31.74.194:38711)
0806 02:58:39.884729 (+ 2846us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:39.884743 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:39.885213 (+ 470us) server_negotiation.cc:407] Connection header received
0806 02:58:39.886007 (+ 794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:39.886020 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:39.886431 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:39.886717 (+ 286us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:58:39.888092 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:39.890659 (+ 2567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:39.895618 (+ 4959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:39.897608 (+ 1990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:39.897810 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:39.898053 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:39.898783 (+ 730us) server_negotiation.cc:299] Negotiation successful
0806 02:58:39.899045 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2620,"thread_start_us":2272,"threads_started":1}
I20260806 02:58:39.900749 1245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:39.880914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.194:38711 (local address 127.0.0.1:40830)
0806 02:58:39.885087 (+ 4173us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:39.885113 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 02:58:39.885670 (+ 557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:58:39.887073 (+ 1403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:58:39.887081 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:58:39.887121 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:58:39.887831 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:39.887849 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:39.890925 (+ 3076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:39.890933 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:39.895294 (+ 4361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:39.895318 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:39.898043 (+ 2725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:39.898052 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:39.898426 (+ 374us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:39.898471 (+ 45us) client_negotiation.cc:769] Sending connection context
0806 02:58:39.899953 (+ 1482us) client_negotiation.cc:240] Negotiation successful
0806 02:58:39.900155 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4026,"mutex_wait_us":28,"thread_start_us":3879,"threads_started":1}
I20260806 02:58:39.903731 867 tablet_service.cc:1467] Processing CreateTablet for tablet 56068cceffa94896a36c0224b9697dc7 (DEFAULT_TABLE table=test-table [id=bbb4ed02c97440428764077ef26a9758]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:58:39.904758 1248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:39.885928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.195:37639 (local address 127.0.0.1:43874)
0806 02:58:39.892614 (+ 6686us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:39.892635 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 02:58:39.892799 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:58:39.894098 (+ 1299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:58:39.894104 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:58:39.894148 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:58:39.894896 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:39.894910 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:39.897500 (+ 2590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:39.897509 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:39.900619 (+ 3110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:39.900641 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:39.903117 (+ 2476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:39.903125 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:39.903861 (+ 736us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:39.903909 (+ 48us) client_negotiation.cc:769] Sending connection context
0806 02:58:39.904027 (+ 118us) client_negotiation.cc:240] Negotiation successful
0806 02:58:39.904205 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6531,"mutex_wait_us":42,"thread_start_us":6324,"threads_started":1}
I20260806 02:58:39.904772 1247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:39.885912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43874 (local address 127.31.74.195:37639)
0806 02:58:39.891818 (+ 5906us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:39.891828 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:39.892749 (+ 921us) server_negotiation.cc:407] Connection header received
0806 02:58:39.893216 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:39.893229 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:39.893720 (+ 491us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:39.894096 (+ 376us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:58:39.895252 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:39.897230 (+ 1978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:39.900938 (+ 3708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:39.902840 (+ 1902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:39.902988 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:39.903250 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:39.904155 (+ 905us) server_negotiation.cc:299] Negotiation successful
0806 02:58:39.904408 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5637,"thread_start_us":5714,"threads_started":1}
I20260806 02:58:39.905685 867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56068cceffa94896a36c0224b9697dc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:58:39.905488 1015 tablet_service.cc:1467] Processing CreateTablet for tablet 56068cceffa94896a36c0224b9697dc7 (DEFAULT_TABLE table=test-table [id=bbb4ed02c97440428764077ef26a9758]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:58:39.907812 1015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56068cceffa94896a36c0224b9697dc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:58:39.909668 1159 tablet_service.cc:1467] Processing CreateTablet for tablet 56068cceffa94896a36c0224b9697dc7 (DEFAULT_TABLE table=test-table [id=bbb4ed02c97440428764077ef26a9758]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:58:39.911623 1159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56068cceffa94896a36c0224b9697dc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:58:39.920935 1249 tablet_bootstrap.cc:492] T 56068cceffa94896a36c0224b9697dc7 P c5b5c4a10bc14bf6b7efe6cbdcfad492: Bootstrap starting.
I20260806 02:58:39.925621 1249 tablet_bootstrap.cc:654] T 56068cceffa94896a36c0224b9697dc7 P c5b5c4a10bc14bf6b7efe6cbdcfad492: Neither blocks nor log segments found. Creating new log.
I20260806 02:58:39.926898 1250 tablet_bootstrap.cc:492] T 56068cceffa94896a36c0224b9697dc7 P 66dd377b30ea48b8a33c9892e715e9f0: Bootstrap starting.
I20260806 02:58:39.927567 1249 log.cc:824] T 56068cceffa94896a36c0224b9697dc7 P c5b5c4a10bc14bf6b7efe6cbdcfad492: Log is configured to *not* fsync() on all Append() calls
I20260806 02:58:39.931335 1250 tablet_bootstrap.cc:654] T 56068cceffa94896a36c0224b9697dc7 P 66dd377b30ea48b8a33c9892e715e9f0: Neither blocks nor log segments found. Creating new log.
I20260806 02:58:39.931739 1252 tablet_bootstrap.cc:492] T 56068cceffa94896a36c0224b9697dc7 P f9b291a1bfb24cb0af222df109aac8ce: Bootstrap starting.
I20260806 02:58:39.932515 1249 tablet_bootstrap.cc:492] T 56068cceffa94896a36c0224b9697dc7 P c5b5c4a10bc14bf6b7efe6cbdcfad492: No bootstrap required, opened a new log
I20260806 02:58:39.932904 1249 ts_tablet_manager.cc:1397] T 56068cceffa94896a36c0224b9697dc7 P c5b5c4a10bc14bf6b7efe6cbdcfad492: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.006s
I20260806 02:58:39.933319 1250 log.cc:824] T 56068cceffa94896a36c0224b9697dc7 P 66dd377b30ea48b8a33c9892e715e9f0: Log is configured to *not* fsync() on all Append() calls
I20260806 02:58:39.935959 1252 tablet_bootstrap.cc:654] T 56068cceffa94896a36c0224b9697dc7 P f9b291a1bfb24cb0af222df109aac8ce: Neither blocks nor log segments found. Creating new log.
I20260806 02:58:39.937507 1250 tablet_bootstrap.cc:492] T 56068cceffa94896a36c0224b9697dc7 P 66dd377b30ea48b8a33c9892e715e9f0: No bootstrap required, opened a new log
I20260806 02:58:39.937505 1252 log.cc:824] T 56068cceffa94896a36c0224b9697dc7 P f9b291a1bfb24cb0af222df109aac8ce: Log is configured to *not* fsync() on all Append() calls
I20260806 02:58:39.938159 1250 ts_tablet_manager.cc:1397] T 56068cceffa94896a36c0224b9697dc7 P 66dd377b30ea48b8a33c9892e715e9f0: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20260806 02:58:39.943073 1252 tablet_bootstrap.cc:492] T 56068cceffa94896a36c0224b9697dc7 P f9b291a1bfb24cb0af222df109aac8ce: No bootstrap required, opened a new log
I20260806 02:58:39.943475 1252 ts_tablet_manager.cc:1397] T 56068cceffa94896a36c0224b9697dc7 P f9b291a1bfb24cb0af222df109aac8ce: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20260806 02:58:39.946313 1249 raft_consensus.cc:348] T 56068cceffa94896a36c0224b9697dc7 P c5b5c4a10bc14bf6b7efe6cbdcfad492 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b5c4a10bc14bf6b7efe6cbdcfad492" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 36295 } } peers { permanent_uuid: "66dd377b30ea48b8a33c9892e715e9f0" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 38711 } } peers { permanent_uuid: "f9b291a1bfb24cb0af222df109aac8ce" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 37639 } }
I20260806 02:58:39.946942 1249 raft_consensus.cc:374] T 56068cceffa94896a36c0224b9697dc7 P c5b5c4a10bc14bf6b7efe6cbdcfad492 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:58:39.947069 1249 raft_consensus.cc:729] T 56068cceffa94896a36c0224b9697dc7 P c5b5c4a10bc14bf6b7efe6cbdcfad492 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5b5c4a10bc14bf6b7efe6cbdcfad492, State: Initialized, Role: FOLLOWER
I20260806 02:58:39.947750 1249 consensus_queue.cc:260] T 56068cceffa94896a36c0224b9697dc7 P c5b5c4a10bc14bf6b7efe6cbdcfad492 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b5c4a10bc14bf6b7efe6cbdcfad492" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 36295 } } peers { permanent_uuid: "66dd377b30ea48b8a33c9892e715e9f0" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 38711 } } peers { permanent_uuid: "f9b291a1bfb24cb0af222df109aac8ce" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 37639 } }
I20260806 02:58:39.949674 1249 ts_tablet_manager.cc:1428] T 56068cceffa94896a36c0224b9697dc7 P c5b5c4a10bc14bf6b7efe6cbdcfad492: Time spent starting tablet: real 0.017s user 0.014s sys 0.001s
I20260806 02:58:39.952809 1250 raft_consensus.cc:348] T 56068cceffa94896a36c0224b9697dc7 P 66dd377b30ea48b8a33c9892e715e9f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b5c4a10bc14bf6b7efe6cbdcfad492" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 36295 } } peers { permanent_uuid: "66dd377b30ea48b8a33c9892e715e9f0" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 38711 } } peers { permanent_uuid: "f9b291a1bfb24cb0af222df109aac8ce" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 37639 } }
I20260806 02:58:39.953361 1250 raft_consensus.cc:374] T 56068cceffa94896a36c0224b9697dc7 P 66dd377b30ea48b8a33c9892e715e9f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:58:39.953459 1250 raft_consensus.cc:729] T 56068cceffa94896a36c0224b9697dc7 P 66dd377b30ea48b8a33c9892e715e9f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66dd377b30ea48b8a33c9892e715e9f0, State: Initialized, Role: FOLLOWER
I20260806 02:58:39.954452 1250 consensus_queue.cc:260] T 56068cceffa94896a36c0224b9697dc7 P 66dd377b30ea48b8a33c9892e715e9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b5c4a10bc14bf6b7efe6cbdcfad492" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 36295 } } peers { permanent_uuid: "66dd377b30ea48b8a33c9892e715e9f0" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 38711 } } peers { permanent_uuid: "f9b291a1bfb24cb0af222df109aac8ce" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 37639 } }
I20260806 02:58:39.956075 1086 heartbeater.cc:503] Master 127.31.74.254:37001 was elected leader, sending a full tablet report...
I20260806 02:58:39.956779 1250 ts_tablet_manager.cc:1428] T 56068cceffa94896a36c0224b9697dc7 P 66dd377b30ea48b8a33c9892e715e9f0: Time spent starting tablet: real 0.018s user 0.018s sys 0.000s
I20260806 02:58:39.957242 1252 raft_consensus.cc:348] T 56068cceffa94896a36c0224b9697dc7 P f9b291a1bfb24cb0af222df109aac8ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b5c4a10bc14bf6b7efe6cbdcfad492" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 36295 } } peers { permanent_uuid: "66dd377b30ea48b8a33c9892e715e9f0" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 38711 } } peers { permanent_uuid: "f9b291a1bfb24cb0af222df109aac8ce" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 37639 } }
I20260806 02:58:39.957734 1252 raft_consensus.cc:374] T 56068cceffa94896a36c0224b9697dc7 P f9b291a1bfb24cb0af222df109aac8ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:58:39.957814 1252 raft_consensus.cc:729] T 56068cceffa94896a36c0224b9697dc7 P f9b291a1bfb24cb0af222df109aac8ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9b291a1bfb24cb0af222df109aac8ce, State: Initialized, Role: FOLLOWER
I20260806 02:58:39.958642 1252 consensus_queue.cc:260] T 56068cceffa94896a36c0224b9697dc7 P f9b291a1bfb24cb0af222df109aac8ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b5c4a10bc14bf6b7efe6cbdcfad492" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 36295 } } peers { permanent_uuid: "66dd377b30ea48b8a33c9892e715e9f0" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 38711 } } peers { permanent_uuid: "f9b291a1bfb24cb0af222df109aac8ce" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 37639 } }
I20260806 02:58:39.960532 1230 heartbeater.cc:503] Master 127.31.74.254:37001 was elected leader, sending a full tablet report...
I20260806 02:58:39.961436 1252 ts_tablet_manager.cc:1428] T 56068cceffa94896a36c0224b9697dc7 P f9b291a1bfb24cb0af222df109aac8ce: Time spent starting tablet: real 0.018s user 0.014s sys 0.003s
W20260806 02:58:40.029268 1231 tablet.cc:2369] T 56068cceffa94896a36c0224b9697dc7 P f9b291a1bfb24cb0af222df109aac8ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:58:40.090878 1087 tablet.cc:2369] T 56068cceffa94896a36c0224b9697dc7 P 66dd377b30ea48b8a33c9892e715e9f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:58:40.173671 944 tablet.cc:2369] T 56068cceffa94896a36c0224b9697dc7 P c5b5c4a10bc14bf6b7efe6cbdcfad492: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:58:40.262709 1257 raft_consensus.cc:482] T 56068cceffa94896a36c0224b9697dc7 P f9b291a1bfb24cb0af222df109aac8ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:58:40.263001 1257 raft_consensus.cc:504] T 56068cceffa94896a36c0224b9697dc7 P f9b291a1bfb24cb0af222df109aac8ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b5c4a10bc14bf6b7efe6cbdcfad492" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 36295 } } peers { permanent_uuid: "66dd377b30ea48b8a33c9892e715e9f0" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 38711 } } peers { permanent_uuid: "f9b291a1bfb24cb0af222df109aac8ce" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 37639 } }
I20260806 02:58:40.265638 1257 leader_election.cc:288] T 56068cceffa94896a36c0224b9697dc7 P f9b291a1bfb24cb0af222df109aac8ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5b5c4a10bc14bf6b7efe6cbdcfad492 (127.31.74.193:36295), 66dd377b30ea48b8a33c9892e715e9f0 (127.31.74.194:38711)
I20260806 02:58:40.277104 1244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:40.266149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.195:58165 (local address 127.31.74.193:36295)
0806 02:58:40.266415 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:40.266422 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:40.267380 (+ 958us) server_negotiation.cc:407] Connection header received
0806 02:58:40.267790 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:40.267797 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:40.267901 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:40.268048 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:58:40.268987 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:40.270490 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:40.274053 (+ 3563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:40.275548 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:40.275804 (+ 256us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:40.275887 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:40.276561 (+ 674us) server_negotiation.cc:299] Negotiation successful
0806 02:58:40.276731 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260806 02:58:40.277109 1246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:40.266245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.195:37367 (local address 127.31.74.194:38711)
0806 02:58:40.266459 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:40.266468 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:40.267327 (+ 859us) server_negotiation.cc:407] Connection header received
0806 02:58:40.268303 (+ 976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:40.268364 (+ 61us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:40.268484 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:40.268611 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:58:40.269332 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:40.270734 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:40.274052 (+ 3318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:40.275548 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:40.275783 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:40.275887 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:40.276561 (+ 674us) server_negotiation.cc:299] Negotiation successful
0806 02:58:40.276734 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 02:58:40.277208 1259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:40.266001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:36295 (local address 127.31.74.195:58165)
0806 02:58:40.267157 (+ 1156us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:40.267182 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 02:58:40.267458 (+ 276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:58:40.268164 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:58:40.268170 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:58:40.268212 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:58:40.268688 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:40.268702 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:40.270761 (+ 2059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:40.270768 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:40.273758 (+ 2990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:40.273786 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:40.275954 (+ 2168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:40.275962 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:40.276272 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:40.276351 (+ 79us) client_negotiation.cc:769] Sending connection context
0806 02:58:40.276469 (+ 118us) client_negotiation.cc:240] Negotiation successful
0806 02:58:40.276672 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":773,"mutex_wait_us":284,"thread_start_us":683,"threads_started":1}
I20260806 02:58:40.277242 1258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:40.266043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.194:38711 (local address 127.31.74.195:37367)
0806 02:58:40.267157 (+ 1114us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:40.267182 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 02:58:40.267440 (+ 258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:58:40.268683 (+ 1243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:58:40.268688 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:58:40.268716 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:58:40.269122 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:40.269131 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:40.270873 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:40.270878 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:40.273756 (+ 2878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:40.273783 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:40.275969 (+ 2186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:40.275975 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:40.276267 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:40.276351 (+ 84us) client_negotiation.cc:769] Sending connection context
0806 02:58:40.276469 (+ 118us) client_negotiation.cc:240] Negotiation successful
0806 02:58:40.276672 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":776,"mutex_wait_us":263,"thread_start_us":565,"threads_started":1}
I20260806 02:58:40.279284 1035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56068cceffa94896a36c0224b9697dc7" candidate_uuid: "f9b291a1bfb24cb0af222df109aac8ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66dd377b30ea48b8a33c9892e715e9f0" is_pre_election: true
I20260806 02:58:40.279284 889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56068cceffa94896a36c0224b9697dc7" candidate_uuid: "f9b291a1bfb24cb0af222df109aac8ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5b5c4a10bc14bf6b7efe6cbdcfad492" is_pre_election: true
I20260806 02:58:40.280194 1035 raft_consensus.cc:2454] T 56068cceffa94896a36c0224b9697dc7 P 66dd377b30ea48b8a33c9892e715e9f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9b291a1bfb24cb0af222df109aac8ce in term 0.
I20260806 02:58:40.280210 889 raft_consensus.cc:2454] T 56068cceffa94896a36c0224b9697dc7 P c5b5c4a10bc14bf6b7efe6cbdcfad492 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9b291a1bfb24cb0af222df109aac8ce in term 0.
I20260806 02:58:40.281458 1108 leader_election.cc:302] T 56068cceffa94896a36c0224b9697dc7 P f9b291a1bfb24cb0af222df109aac8ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66dd377b30ea48b8a33c9892e715e9f0, f9b291a1bfb24cb0af222df109aac8ce; no voters:
I20260806 02:58:40.282089 1257 raft_consensus.cc:2781] T 56068cceffa94896a36c0224b9697dc7 P f9b291a1bfb24cb0af222df109aac8ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:58:40.282370 1257 raft_consensus.cc:482] T 56068cceffa94896a36c0224b9697dc7 P f9b291a1bfb24cb0af222df109aac8ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:58:40.282541 1257 raft_consensus.cc:3037] T 56068cceffa94896a36c0224b9697dc7 P f9b291a1bfb24cb0af222df109aac8ce [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:58:40.286268 1257 raft_consensus.cc:504] T 56068cceffa94896a36c0224b9697dc7 P f9b291a1bfb24cb0af222df109aac8ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b5c4a10bc14bf6b7efe6cbdcfad492" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 36295 } } peers { permanent_uuid: "66dd377b30ea48b8a33c9892e715e9f0" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 38711 } } peers { permanent_uuid: "f9b291a1bfb24cb0af222df109aac8ce" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 37639 } }
I20260806 02:58:40.287117 1257 leader_election.cc:288] T 56068cceffa94896a36c0224b9697dc7 P f9b291a1bfb24cb0af222df109aac8ce [CANDIDATE]: Term 1 election: Requested vote from peers c5b5c4a10bc14bf6b7efe6cbdcfad492 (127.31.74.193:36295), 66dd377b30ea48b8a33c9892e715e9f0 (127.31.74.194:38711)
I20260806 02:58:40.287869 1035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56068cceffa94896a36c0224b9697dc7" candidate_uuid: "f9b291a1bfb24cb0af222df109aac8ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66dd377b30ea48b8a33c9892e715e9f0"
I20260806 02:58:40.287968 889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56068cceffa94896a36c0224b9697dc7" candidate_uuid: "f9b291a1bfb24cb0af222df109aac8ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5b5c4a10bc14bf6b7efe6cbdcfad492"
I20260806 02:58:40.288158 1035 raft_consensus.cc:3037] T 56068cceffa94896a36c0224b9697dc7 P 66dd377b30ea48b8a33c9892e715e9f0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:58:40.288321 889 raft_consensus.cc:3037] T 56068cceffa94896a36c0224b9697dc7 P c5b5c4a10bc14bf6b7efe6cbdcfad492 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:58:40.291245 1035 raft_consensus.cc:2454] T 56068cceffa94896a36c0224b9697dc7 P 66dd377b30ea48b8a33c9892e715e9f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9b291a1bfb24cb0af222df109aac8ce in term 1.
I20260806 02:58:40.291393 889 raft_consensus.cc:2454] T 56068cceffa94896a36c0224b9697dc7 P c5b5c4a10bc14bf6b7efe6cbdcfad492 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9b291a1bfb24cb0af222df109aac8ce in term 1.
I20260806 02:58:40.292073 1108 leader_election.cc:302] T 56068cceffa94896a36c0224b9697dc7 P f9b291a1bfb24cb0af222df109aac8ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66dd377b30ea48b8a33c9892e715e9f0, f9b291a1bfb24cb0af222df109aac8ce; no voters:
I20260806 02:58:40.292804 1257 raft_consensus.cc:2781] T 56068cceffa94896a36c0224b9697dc7 P f9b291a1bfb24cb0af222df109aac8ce [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:58:40.293614 1257 raft_consensus.cc:686] T 56068cceffa94896a36c0224b9697dc7 P f9b291a1bfb24cb0af222df109aac8ce [term 1 LEADER]: Becoming Leader. State: Replica: f9b291a1bfb24cb0af222df109aac8ce, State: Running, Role: LEADER
I20260806 02:58:40.294154 1257 consensus_queue.cc:237] T 56068cceffa94896a36c0224b9697dc7 P f9b291a1bfb24cb0af222df109aac8ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b5c4a10bc14bf6b7efe6cbdcfad492" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 36295 } } peers { permanent_uuid: "66dd377b30ea48b8a33c9892e715e9f0" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 38711 } } peers { permanent_uuid: "f9b291a1bfb24cb0af222df109aac8ce" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 37639 } }
I20260806 02:58:40.300374 726 catalog_manager.cc:5495] T 56068cceffa94896a36c0224b9697dc7 P f9b291a1bfb24cb0af222df109aac8ce reported cstate change: term changed from 0 to 1, leader changed from <none> to f9b291a1bfb24cb0af222df109aac8ce (127.31.74.195). New cstate: current_term: 1 leader_uuid: "f9b291a1bfb24cb0af222df109aac8ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b5c4a10bc14bf6b7efe6cbdcfad492" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 36295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66dd377b30ea48b8a33c9892e715e9f0" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 38711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9b291a1bfb24cb0af222df109aac8ce" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 37639 } health_report { overall_health: HEALTHY } } }
I20260806 02:58:40.373476 1247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:40.363415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43882 (local address 127.31.74.195:37639)
0806 02:58:40.363712 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:40.363718 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:40.364247 (+ 529us) server_negotiation.cc:407] Connection header received
0806 02:58:40.364514 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:40.364519 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:40.364613 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:40.364931 (+ 318us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:58:40.365695 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:40.367169 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:40.370408 (+ 3239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:40.371278 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:40.371443 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:40.372171 (+ 728us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:58:40.372776 (+ 605us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:58:40.372904 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:40.373131 (+ 227us) server_negotiation.cc:299] Negotiation successful
0806 02:58:40.373219 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260806 02:58:40.382488 889 raft_consensus.cc:1261] T 56068cceffa94896a36c0224b9697dc7 P c5b5c4a10bc14bf6b7efe6cbdcfad492 [term 1 FOLLOWER]: Refusing update from remote peer f9b291a1bfb24cb0af222df109aac8ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:58:40.382504 1035 raft_consensus.cc:1261] T 56068cceffa94896a36c0224b9697dc7 P 66dd377b30ea48b8a33c9892e715e9f0 [term 1 FOLLOWER]: Refusing update from remote peer f9b291a1bfb24cb0af222df109aac8ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:58:40.383653 1260 consensus_queue.cc:1035] T 56068cceffa94896a36c0224b9697dc7 P f9b291a1bfb24cb0af222df109aac8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5b5c4a10bc14bf6b7efe6cbdcfad492" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 36295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:58:40.384045 1257 consensus_queue.cc:1035] T 56068cceffa94896a36c0224b9697dc7 P f9b291a1bfb24cb0af222df109aac8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "66dd377b30ea48b8a33c9892e715e9f0" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 38711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:58:40.402169 1264 mvcc.cc:205] Tried to move back new op lower bound from 7315395053075333120 to 7315395052728991744. Current Snapshot: MvccSnapshot[applied={T|T < 7315395053075333120}]
I20260806 02:58:40.429991 726 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:58:40.430437 726 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:58:40.435145 726 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b5e6b82242fc46519f1bfc172a68c982: Sending DeleteTablet for 3 replicas of tablet 56068cceffa94896a36c0224b9697dc7
I20260806 02:58:40.436594 1015 tablet_service.cc:1514] Processing DeleteTablet for tablet 56068cceffa94896a36c0224b9697dc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:58:40 UTC) from {username='kudu', principal='kudu/127.31.74.254@KRBTEST.COM'} at 127.0.0.1:40830
I20260806 02:58:40.436653 867 tablet_service.cc:1514] Processing DeleteTablet for tablet 56068cceffa94896a36c0224b9697dc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:58:40 UTC) from {username='kudu', principal='kudu/127.31.74.254@KRBTEST.COM'} at 127.0.0.1:49994
I20260806 02:58:40.437458 1275 tablet_replica.cc:330] T 56068cceffa94896a36c0224b9697dc7 P 66dd377b30ea48b8a33c9892e715e9f0: stopping tablet replica
I20260806 02:58:40.437970 1275 raft_consensus.cc:2229] T 56068cceffa94896a36c0224b9697dc7 P 66dd377b30ea48b8a33c9892e715e9f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:58:40.438668 1159 tablet_service.cc:1514] Processing DeleteTablet for tablet 56068cceffa94896a36c0224b9697dc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:58:40 UTC) from {username='kudu', principal='kudu/127.31.74.254@KRBTEST.COM'} at 127.0.0.1:43874
I20260806 02:58:40.438815 1275 raft_consensus.cc:2258] T 56068cceffa94896a36c0224b9697dc7 P 66dd377b30ea48b8a33c9892e715e9f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:58:40.442898 726 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39804:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:58:40.444072 726 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:58:40.444710 1275 ts_tablet_manager.cc:1905] T 56068cceffa94896a36c0224b9697dc7 P 66dd377b30ea48b8a33c9892e715e9f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:58:40.447394 1276 tablet_replica.cc:330] T 56068cceffa94896a36c0224b9697dc7 P c5b5c4a10bc14bf6b7efe6cbdcfad492: stopping tablet replica
I20260806 02:58:40.448099 1277 tablet_replica.cc:330] T 56068cceffa94896a36c0224b9697dc7 P f9b291a1bfb24cb0af222df109aac8ce: stopping tablet replica
I20260806 02:58:40.448696 1277 raft_consensus.cc:2229] T 56068cceffa94896a36c0224b9697dc7 P f9b291a1bfb24cb0af222df109aac8ce [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:58:40.449209 1276 raft_consensus.cc:2229] T 56068cceffa94896a36c0224b9697dc7 P c5b5c4a10bc14bf6b7efe6cbdcfad492 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:58:40.450095 1277 raft_consensus.cc:2258] T 56068cceffa94896a36c0224b9697dc7 P f9b291a1bfb24cb0af222df109aac8ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:58:40.450718 1276 raft_consensus.cc:2258] T 56068cceffa94896a36c0224b9697dc7 P c5b5c4a10bc14bf6b7efe6cbdcfad492 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:58:40.453361 1276 ts_tablet_manager.cc:1905] T 56068cceffa94896a36c0224b9697dc7 P c5b5c4a10bc14bf6b7efe6cbdcfad492: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:58:40.454107 1275 ts_tablet_manager.cc:1918] T 56068cceffa94896a36c0224b9697dc7 P 66dd377b30ea48b8a33c9892e715e9f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:58:40.454337 1275 log.cc:1196] T 56068cceffa94896a36c0224b9697dc7 P 66dd377b30ea48b8a33c9892e715e9f0: Deleting WAL directory at /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/minicluster-data/ts-1/wal/wals/56068cceffa94896a36c0224b9697dc7
I20260806 02:58:40.455008 1275 ts_tablet_manager.cc:1939] T 56068cceffa94896a36c0224b9697dc7 P 66dd377b30ea48b8a33c9892e715e9f0: Deleting consensus metadata
I20260806 02:58:40.457247 710 catalog_manager.cc:4841] TS 66dd377b30ea48b8a33c9892e715e9f0 (127.31.74.194:38711): tablet 56068cceffa94896a36c0224b9697dc7 (table test-table [id=bbb4ed02c97440428764077ef26a9758]) successfully deleted
I20260806 02:58:40.457376 1015 tablet_service.cc:1467] Processing CreateTablet for tablet e7f0f8fdbb384284931e197973452a79 (DEFAULT_TABLE table=test-table [id=319d99217062406c9cee71c4b0ded160]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:58:40.457867 1015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7f0f8fdbb384284931e197973452a79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:58:40.458964 867 tablet_service.cc:1467] Processing CreateTablet for tablet e7f0f8fdbb384284931e197973452a79 (DEFAULT_TABLE table=test-table [id=319d99217062406c9cee71c4b0ded160]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:58:40.459502 867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7f0f8fdbb384284931e197973452a79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:58:40.462106 1276 ts_tablet_manager.cc:1918] T 56068cceffa94896a36c0224b9697dc7 P c5b5c4a10bc14bf6b7efe6cbdcfad492: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:58:40.462313 1276 log.cc:1196] T 56068cceffa94896a36c0224b9697dc7 P c5b5c4a10bc14bf6b7efe6cbdcfad492: Deleting WAL directory at /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/minicluster-data/ts-0/wal/wals/56068cceffa94896a36c0224b9697dc7
I20260806 02:58:40.463018 1276 ts_tablet_manager.cc:1939] T 56068cceffa94896a36c0224b9697dc7 P c5b5c4a10bc14bf6b7efe6cbdcfad492: Deleting consensus metadata
I20260806 02:58:40.464368 1159 tablet_service.cc:1467] Processing CreateTablet for tablet e7f0f8fdbb384284931e197973452a79 (DEFAULT_TABLE table=test-table [id=319d99217062406c9cee71c4b0ded160]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:58:40.465164 1159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7f0f8fdbb384284931e197973452a79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:58:40.465415 711 catalog_manager.cc:4841] TS c5b5c4a10bc14bf6b7efe6cbdcfad492 (127.31.74.193:36295): tablet 56068cceffa94896a36c0224b9697dc7 (table test-table [id=bbb4ed02c97440428764077ef26a9758]) successfully deleted
I20260806 02:58:40.475356 1277 ts_tablet_manager.cc:1905] T 56068cceffa94896a36c0224b9697dc7 P f9b291a1bfb24cb0af222df109aac8ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:58:40.478664 1278 tablet_bootstrap.cc:492] T e7f0f8fdbb384284931e197973452a79 P 66dd377b30ea48b8a33c9892e715e9f0: Bootstrap starting.
I20260806 02:58:40.481171 1278 tablet_bootstrap.cc:654] T e7f0f8fdbb384284931e197973452a79 P 66dd377b30ea48b8a33c9892e715e9f0: Neither blocks nor log segments found. Creating new log.
I20260806 02:58:40.482848 1279 tablet_bootstrap.cc:492] T e7f0f8fdbb384284931e197973452a79 P c5b5c4a10bc14bf6b7efe6cbdcfad492: Bootstrap starting.
I20260806 02:58:40.485494 1278 tablet_bootstrap.cc:492] T e7f0f8fdbb384284931e197973452a79 P 66dd377b30ea48b8a33c9892e715e9f0: No bootstrap required, opened a new log
I20260806 02:58:40.485749 1278 ts_tablet_manager.cc:1397] T e7f0f8fdbb384284931e197973452a79 P 66dd377b30ea48b8a33c9892e715e9f0: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20260806 02:58:40.486848 1278 raft_consensus.cc:348] T e7f0f8fdbb384284931e197973452a79 P 66dd377b30ea48b8a33c9892e715e9f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b5c4a10bc14bf6b7efe6cbdcfad492" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 36295 } } peers { permanent_uuid: "66dd377b30ea48b8a33c9892e715e9f0" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 38711 } } peers { permanent_uuid: "f9b291a1bfb24cb0af222df109aac8ce" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 37639 } }
I20260806 02:58:40.487143 1278 raft_consensus.cc:374] T e7f0f8fdbb384284931e197973452a79 P 66dd377b30ea48b8a33c9892e715e9f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:58:40.487242 1278 raft_consensus.cc:729] T e7f0f8fdbb384284931e197973452a79 P 66dd377b30ea48b8a33c9892e715e9f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66dd377b30ea48b8a33c9892e715e9f0, State: Initialized, Role: FOLLOWER
I20260806 02:58:40.487527 1278 consensus_queue.cc:260] T e7f0f8fdbb384284931e197973452a79 P 66dd377b30ea48b8a33c9892e715e9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b5c4a10bc14bf6b7efe6cbdcfad492" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 36295 } } peers { permanent_uuid: "66dd377b30ea48b8a33c9892e715e9f0" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 38711 } } peers { permanent_uuid: "f9b291a1bfb24cb0af222df109aac8ce" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 37639 } }
I20260806 02:58:40.487941 1279 tablet_bootstrap.cc:654] T e7f0f8fdbb384284931e197973452a79 P c5b5c4a10bc14bf6b7efe6cbdcfad492: Neither blocks nor log segments found. Creating new log.
I20260806 02:58:40.488462 1278 ts_tablet_manager.cc:1428] T e7f0f8fdbb384284931e197973452a79 P 66dd377b30ea48b8a33c9892e715e9f0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260806 02:58:40.492903 1277 ts_tablet_manager.cc:1918] T 56068cceffa94896a36c0224b9697dc7 P f9b291a1bfb24cb0af222df109aac8ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:58:40.493306 1277 log.cc:1196] T 56068cceffa94896a36c0224b9697dc7 P f9b291a1bfb24cb0af222df109aac8ce: Deleting WAL directory at /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/minicluster-data/ts-2/wal/wals/56068cceffa94896a36c0224b9697dc7
I20260806 02:58:40.493047 1282 tablet_bootstrap.cc:492] T e7f0f8fdbb384284931e197973452a79 P f9b291a1bfb24cb0af222df109aac8ce: Bootstrap starting.
I20260806 02:58:40.494486 1277 ts_tablet_manager.cc:1939] T 56068cceffa94896a36c0224b9697dc7 P f9b291a1bfb24cb0af222df109aac8ce: Deleting consensus metadata
I20260806 02:58:40.496228 1282 tablet_bootstrap.cc:654] T e7f0f8fdbb384284931e197973452a79 P f9b291a1bfb24cb0af222df109aac8ce: Neither blocks nor log segments found. Creating new log.
I20260806 02:58:40.496752 710 catalog_manager.cc:4841] TS f9b291a1bfb24cb0af222df109aac8ce (127.31.74.195:37639): tablet 56068cceffa94896a36c0224b9697dc7 (table test-table [id=bbb4ed02c97440428764077ef26a9758]) successfully deleted
I20260806 02:58:40.500310 1279 tablet_bootstrap.cc:492] T e7f0f8fdbb384284931e197973452a79 P c5b5c4a10bc14bf6b7efe6cbdcfad492: No bootstrap required, opened a new log
I20260806 02:58:40.500515 1279 ts_tablet_manager.cc:1397] T e7f0f8fdbb384284931e197973452a79 P c5b5c4a10bc14bf6b7efe6cbdcfad492: Time spent bootstrapping tablet: real 0.018s user 0.010s sys 0.000s
I20260806 02:58:40.501438 1279 raft_consensus.cc:348] T e7f0f8fdbb384284931e197973452a79 P c5b5c4a10bc14bf6b7efe6cbdcfad492 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b5c4a10bc14bf6b7efe6cbdcfad492" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 36295 } } peers { permanent_uuid: "66dd377b30ea48b8a33c9892e715e9f0" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 38711 } } peers { permanent_uuid: "f9b291a1bfb24cb0af222df109aac8ce" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 37639 } }
I20260806 02:58:40.501761 1279 raft_consensus.cc:374] T e7f0f8fdbb384284931e197973452a79 P c5b5c4a10bc14bf6b7efe6cbdcfad492 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:58:40.501878 1279 raft_consensus.cc:729] T e7f0f8fdbb384284931e197973452a79 P c5b5c4a10bc14bf6b7efe6cbdcfad492 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5b5c4a10bc14bf6b7efe6cbdcfad492, State: Initialized, Role: FOLLOWER
I20260806 02:58:40.502162 1279 consensus_queue.cc:260] T e7f0f8fdbb384284931e197973452a79 P c5b5c4a10bc14bf6b7efe6cbdcfad492 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b5c4a10bc14bf6b7efe6cbdcfad492" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 36295 } } peers { permanent_uuid: "66dd377b30ea48b8a33c9892e715e9f0" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 38711 } } peers { permanent_uuid: "f9b291a1bfb24cb0af222df109aac8ce" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 37639 } }
I20260806 02:58:40.502864 1279 ts_tablet_manager.cc:1428] T e7f0f8fdbb384284931e197973452a79 P c5b5c4a10bc14bf6b7efe6cbdcfad492: Time spent starting tablet: real 0.002s user 0.001s sys 0.003s
I20260806 02:58:40.506740 1282 tablet_bootstrap.cc:492] T e7f0f8fdbb384284931e197973452a79 P f9b291a1bfb24cb0af222df109aac8ce: No bootstrap required, opened a new log
I20260806 02:58:40.506937 1282 ts_tablet_manager.cc:1397] T e7f0f8fdbb384284931e197973452a79 P f9b291a1bfb24cb0af222df109aac8ce: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.000s
I20260806 02:58:40.507755 1282 raft_consensus.cc:348] T e7f0f8fdbb384284931e197973452a79 P f9b291a1bfb24cb0af222df109aac8ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b5c4a10bc14bf6b7efe6cbdcfad492" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 36295 } } peers { permanent_uuid: "66dd377b30ea48b8a33c9892e715e9f0" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 38711 } } peers { permanent_uuid: "f9b291a1bfb24cb0af222df109aac8ce" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 37639 } }
I20260806 02:58:40.508036 1282 raft_consensus.cc:374] T e7f0f8fdbb384284931e197973452a79 P f9b291a1bfb24cb0af222df109aac8ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:58:40.508165 1282 raft_consensus.cc:729] T e7f0f8fdbb384284931e197973452a79 P f9b291a1bfb24cb0af222df109aac8ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9b291a1bfb24cb0af222df109aac8ce, State: Initialized, Role: FOLLOWER
I20260806 02:58:40.508491 1282 consensus_queue.cc:260] T e7f0f8fdbb384284931e197973452a79 P f9b291a1bfb24cb0af222df109aac8ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b5c4a10bc14bf6b7efe6cbdcfad492" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 36295 } } peers { permanent_uuid: "66dd377b30ea48b8a33c9892e715e9f0" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 38711 } } peers { permanent_uuid: "f9b291a1bfb24cb0af222df109aac8ce" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 37639 } }
I20260806 02:58:40.510517 1282 ts_tablet_manager.cc:1428] T e7f0f8fdbb384284931e197973452a79 P f9b291a1bfb24cb0af222df109aac8ce: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260806 02:58:40.517310 1255 raft_consensus.cc:482] T e7f0f8fdbb384284931e197973452a79 P c5b5c4a10bc14bf6b7efe6cbdcfad492 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:58:40.517746 1255 raft_consensus.cc:504] T e7f0f8fdbb384284931e197973452a79 P c5b5c4a10bc14bf6b7efe6cbdcfad492 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b5c4a10bc14bf6b7efe6cbdcfad492" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 36295 } } peers { permanent_uuid: "66dd377b30ea48b8a33c9892e715e9f0" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 38711 } } peers { permanent_uuid: "f9b291a1bfb24cb0af222df109aac8ce" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 37639 } }
I20260806 02:58:40.519740 1255 leader_election.cc:288] T e7f0f8fdbb384284931e197973452a79 P c5b5c4a10bc14bf6b7efe6cbdcfad492 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66dd377b30ea48b8a33c9892e715e9f0 (127.31.74.194:38711), f9b291a1bfb24cb0af222df109aac8ce (127.31.74.195:37639)
I20260806 02:58:40.529592 1246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:40.520229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.193:43045 (local address 127.31.74.194:38711)
0806 02:58:40.520441 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:40.520448 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:40.521046 (+ 598us) server_negotiation.cc:407] Connection header received
0806 02:58:40.521134 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:40.521139 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:40.521220 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:40.521330 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:58:40.522378 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:40.523297 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:40.526516 (+ 3219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:40.528162 (+ 1646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:40.528441 (+ 279us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:40.528542 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:40.529071 (+ 529us) server_negotiation.cc:299] Negotiation successful
0806 02:58:40.529247 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 02:58:40.530069 1247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:40.521597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.193:60357 (local address 127.31.74.195:37639)
0806 02:58:40.521803 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:40.521809 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:40.521828 (+ 19us) server_negotiation.cc:407] Connection header received
0806 02:58:40.521899 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:40.521902 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:40.521973 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:40.522068 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:58:40.522796 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:40.524263 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:40.527493 (+ 3230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:40.528755 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:40.528956 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:40.529055 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:40.529543 (+ 488us) server_negotiation.cc:299] Negotiation successful
0806 02:58:40.529711 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 02:58:40.530524 1284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:40.520119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.194:38711 (local address 127.31.74.193:43045)
0806 02:58:40.520826 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:40.520848 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 02:58:40.520993 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:58:40.521745 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:58:40.521751 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:58:40.521782 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:58:40.522163 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:40.522178 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:40.523590 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:40.523600 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:40.526190 (+ 2590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:40.526219 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:40.528496 (+ 2277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:40.528504 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:40.528816 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:40.528872 (+ 56us) client_negotiation.cc:769] Sending connection context
0806 02:58:40.529841 (+ 969us) client_negotiation.cc:240] Negotiation successful
0806 02:58:40.530062 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"mutex_wait_us":1,"thread_start_us":403,"threads_started":1}
I20260806 02:58:40.531118 1285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:40.520982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.195:37639 (local address 127.31.74.193:60357)
0806 02:58:40.521601 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:40.521615 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:58:40.521743 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:58:40.522089 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:58:40.522093 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:58:40.522120 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:58:40.522618 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:40.522631 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:40.524542 (+ 1911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:40.524549 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:40.527241 (+ 2692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:40.527257 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:40.529043 (+ 1786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:40.529051 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:40.529282 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:40.529355 (+ 73us) client_negotiation.cc:769] Sending connection context
0806 02:58:40.530468 (+ 1113us) client_negotiation.cc:240] Negotiation successful
0806 02:58:40.530686 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"mutex_wait_us":34,"thread_start_us":353,"threads_started":1}
I20260806 02:58:40.531528 1035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7f0f8fdbb384284931e197973452a79" candidate_uuid: "c5b5c4a10bc14bf6b7efe6cbdcfad492" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66dd377b30ea48b8a33c9892e715e9f0" is_pre_election: true
I20260806 02:58:40.531832 1035 raft_consensus.cc:2454] T e7f0f8fdbb384284931e197973452a79 P 66dd377b30ea48b8a33c9892e715e9f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5b5c4a10bc14bf6b7efe6cbdcfad492 in term 0.
I20260806 02:58:40.532500 817 leader_election.cc:302] T e7f0f8fdbb384284931e197973452a79 P c5b5c4a10bc14bf6b7efe6cbdcfad492 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66dd377b30ea48b8a33c9892e715e9f0, c5b5c4a10bc14bf6b7efe6cbdcfad492; no voters:
I20260806 02:58:40.533051 1255 raft_consensus.cc:2781] T e7f0f8fdbb384284931e197973452a79 P c5b5c4a10bc14bf6b7efe6cbdcfad492 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:58:40.532820 1179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7f0f8fdbb384284931e197973452a79" candidate_uuid: "c5b5c4a10bc14bf6b7efe6cbdcfad492" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9b291a1bfb24cb0af222df109aac8ce" is_pre_election: true
I20260806 02:58:40.533239 1255 raft_consensus.cc:482] T e7f0f8fdbb384284931e197973452a79 P c5b5c4a10bc14bf6b7efe6cbdcfad492 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:58:40.533351 1255 raft_consensus.cc:3037] T e7f0f8fdbb384284931e197973452a79 P c5b5c4a10bc14bf6b7efe6cbdcfad492 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:58:40.533404 1179 raft_consensus.cc:2454] T e7f0f8fdbb384284931e197973452a79 P f9b291a1bfb24cb0af222df109aac8ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5b5c4a10bc14bf6b7efe6cbdcfad492 in term 0.
I20260806 02:58:40.536855 1255 raft_consensus.cc:504] T e7f0f8fdbb384284931e197973452a79 P c5b5c4a10bc14bf6b7efe6cbdcfad492 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b5c4a10bc14bf6b7efe6cbdcfad492" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 36295 } } peers { permanent_uuid: "66dd377b30ea48b8a33c9892e715e9f0" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 38711 } } peers { permanent_uuid: "f9b291a1bfb24cb0af222df109aac8ce" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 37639 } }
I20260806 02:58:40.537715 1255 leader_election.cc:288] T e7f0f8fdbb384284931e197973452a79 P c5b5c4a10bc14bf6b7efe6cbdcfad492 [CANDIDATE]: Term 1 election: Requested vote from peers 66dd377b30ea48b8a33c9892e715e9f0 (127.31.74.194:38711), f9b291a1bfb24cb0af222df109aac8ce (127.31.74.195:37639)
I20260806 02:58:40.538353 1035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7f0f8fdbb384284931e197973452a79" candidate_uuid: "c5b5c4a10bc14bf6b7efe6cbdcfad492" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66dd377b30ea48b8a33c9892e715e9f0"
I20260806 02:58:40.538522 1179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7f0f8fdbb384284931e197973452a79" candidate_uuid: "c5b5c4a10bc14bf6b7efe6cbdcfad492" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9b291a1bfb24cb0af222df109aac8ce"
I20260806 02:58:40.538628 1035 raft_consensus.cc:3037] T e7f0f8fdbb384284931e197973452a79 P 66dd377b30ea48b8a33c9892e715e9f0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:58:40.538775 1179 raft_consensus.cc:3037] T e7f0f8fdbb384284931e197973452a79 P f9b291a1bfb24cb0af222df109aac8ce [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:58:40.541855 1035 raft_consensus.cc:2454] T e7f0f8fdbb384284931e197973452a79 P 66dd377b30ea48b8a33c9892e715e9f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5b5c4a10bc14bf6b7efe6cbdcfad492 in term 1.
I20260806 02:58:40.542186 1179 raft_consensus.cc:2454] T e7f0f8fdbb384284931e197973452a79 P f9b291a1bfb24cb0af222df109aac8ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5b5c4a10bc14bf6b7efe6cbdcfad492 in term 1.
I20260806 02:58:40.542389 817 leader_election.cc:302] T e7f0f8fdbb384284931e197973452a79 P c5b5c4a10bc14bf6b7efe6cbdcfad492 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66dd377b30ea48b8a33c9892e715e9f0, c5b5c4a10bc14bf6b7efe6cbdcfad492; no voters:
I20260806 02:58:40.542824 1255 raft_consensus.cc:2781] T e7f0f8fdbb384284931e197973452a79 P c5b5c4a10bc14bf6b7efe6cbdcfad492 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:58:40.543681 1255 raft_consensus.cc:686] T e7f0f8fdbb384284931e197973452a79 P c5b5c4a10bc14bf6b7efe6cbdcfad492 [term 1 LEADER]: Becoming Leader. State: Replica: c5b5c4a10bc14bf6b7efe6cbdcfad492, State: Running, Role: LEADER
I20260806 02:58:40.544340 1255 consensus_queue.cc:237] T e7f0f8fdbb384284931e197973452a79 P c5b5c4a10bc14bf6b7efe6cbdcfad492 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b5c4a10bc14bf6b7efe6cbdcfad492" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 36295 } } peers { permanent_uuid: "66dd377b30ea48b8a33c9892e715e9f0" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 38711 } } peers { permanent_uuid: "f9b291a1bfb24cb0af222df109aac8ce" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 37639 } }
I20260806 02:58:40.549043 726 catalog_manager.cc:5495] T e7f0f8fdbb384284931e197973452a79 P c5b5c4a10bc14bf6b7efe6cbdcfad492 reported cstate change: term changed from 0 to 1, leader changed from <none> to c5b5c4a10bc14bf6b7efe6cbdcfad492 (127.31.74.193). New cstate: current_term: 1 leader_uuid: "c5b5c4a10bc14bf6b7efe6cbdcfad492" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b5c4a10bc14bf6b7efe6cbdcfad492" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 36295 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66dd377b30ea48b8a33c9892e715e9f0" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 38711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9b291a1bfb24cb0af222df109aac8ce" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 37639 } health_report { overall_health: UNKNOWN } } }
Aug 06 02:58:40 dist-test-slave-dhph krb5kdc[680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985117, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.254@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 02:58:40.626528 1295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:40.606746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:37001 (local address 127.0.0.1:39806)
0806 02:58:40.607558 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:40.607584 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 02:58:40.607851 (+ 267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:58:40.608398 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:58:40.608403 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:58:40.609488 (+ 1085us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:58:40.609832 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:40.609848 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:40.611701 (+ 1853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:40.611708 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:40.613038 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:40.613057 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:40.614309 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:40.614315 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:40.614476 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:40.615428 (+ 952us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:58:40.615449 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:58:40.618249 (+ 2800us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:58:40.621818 (+ 3569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:58:40.621836 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:58:40.621845 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:58:40.622945 (+ 1100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:58:40.623733 (+ 788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:58:40.623740 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:58:40.623744 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:58:40.623983 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:58:40.624926 (+ 943us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:58:40.624937 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:58:40.625192 (+ 255us) client_negotiation.cc:769] Sending connection context
0806 02:58:40.625565 (+ 373us) client_negotiation.cc:240] Negotiation successful
0806 02:58:40.626040 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"thread_start_us":447,"threads_started":1}
I20260806 02:58:40.626995 1294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:40.606536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39806 (local address 127.31.74.254:37001)
0806 02:58:40.607382 (+ 846us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:40.607393 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:40.607970 (+ 577us) server_negotiation.cc:407] Connection header received
0806 02:58:40.608024 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:40.608028 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:40.608089 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:40.608173 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:40.610032 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:40.611398 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:40.613231 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:40.614073 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:40.614242 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:40.618527 (+ 4285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:58:40.618533 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:58:40.618537 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:58:40.618570 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:58:40.621600 (+ 3030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:40.623113 (+ 1513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:40.623122 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:40.623128 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:40.623482 (+ 354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:40.624208 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:40.624213 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:40.624215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:40.624755 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:58:40.624892 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:40.625743 (+ 851us) server_negotiation.cc:299] Negotiation successful
0806 02:58:40.626394 (+ 651us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":273,"server-negotiator.queue_time_us":656,"thread_start_us":395,"threads_started":1}
I20260806 02:58:40.633441 726 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.31.74.254@KRBTEST.COM'} at 127.0.0.1:39806:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260806 02:58:40.634672 726 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:58:40.642848 1015 tablet_service.cc:1467] Processing CreateTablet for tablet bb728e35dcd047adaaeda5b645ded748 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d28c020eb4c04f48894eabd721e1c7bb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 02:58:40.642849 1159 tablet_service.cc:1467] Processing CreateTablet for tablet bb728e35dcd047adaaeda5b645ded748 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d28c020eb4c04f48894eabd721e1c7bb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 02:58:40.642848 867 tablet_service.cc:1467] Processing CreateTablet for tablet bb728e35dcd047adaaeda5b645ded748 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d28c020eb4c04f48894eabd721e1c7bb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 02:58:40.643460 1015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb728e35dcd047adaaeda5b645ded748. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:58:40.643473 1159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb728e35dcd047adaaeda5b645ded748. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:58:40.643473 867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb728e35dcd047adaaeda5b645ded748. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:58:40.651093 1279 tablet_bootstrap.cc:492] T bb728e35dcd047adaaeda5b645ded748 P c5b5c4a10bc14bf6b7efe6cbdcfad492: Bootstrap starting.
I20260806 02:58:40.653435 1279 tablet_bootstrap.cc:654] T bb728e35dcd047adaaeda5b645ded748 P c5b5c4a10bc14bf6b7efe6cbdcfad492: Neither blocks nor log segments found. Creating new log.
I20260806 02:58:40.653635 1282 tablet_bootstrap.cc:492] T bb728e35dcd047adaaeda5b645ded748 P f9b291a1bfb24cb0af222df109aac8ce: Bootstrap starting.
I20260806 02:58:40.656658 1282 tablet_bootstrap.cc:654] T bb728e35dcd047adaaeda5b645ded748 P f9b291a1bfb24cb0af222df109aac8ce: Neither blocks nor log segments found. Creating new log.
I20260806 02:58:40.656991 1278 tablet_bootstrap.cc:492] T bb728e35dcd047adaaeda5b645ded748 P 66dd377b30ea48b8a33c9892e715e9f0: Bootstrap starting.
I20260806 02:58:40.658905 1278 tablet_bootstrap.cc:654] T bb728e35dcd047adaaeda5b645ded748 P 66dd377b30ea48b8a33c9892e715e9f0: Neither blocks nor log segments found. Creating new log.
I20260806 02:58:40.664517 1282 tablet_bootstrap.cc:492] T bb728e35dcd047adaaeda5b645ded748 P f9b291a1bfb24cb0af222df109aac8ce: No bootstrap required, opened a new log
I20260806 02:58:40.664825 1279 tablet_bootstrap.cc:492] T bb728e35dcd047adaaeda5b645ded748 P c5b5c4a10bc14bf6b7efe6cbdcfad492: No bootstrap required, opened a new log
I20260806 02:58:40.664844 1282 ts_tablet_manager.cc:1397] T bb728e35dcd047adaaeda5b645ded748 P f9b291a1bfb24cb0af222df109aac8ce: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20260806 02:58:40.665095 1279 ts_tablet_manager.cc:1397] T bb728e35dcd047adaaeda5b645ded748 P c5b5c4a10bc14bf6b7efe6cbdcfad492: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.000s
I20260806 02:58:40.665915 1279 raft_consensus.cc:348] T bb728e35dcd047adaaeda5b645ded748 P c5b5c4a10bc14bf6b7efe6cbdcfad492 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b5c4a10bc14bf6b7efe6cbdcfad492" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 36295 } } peers { permanent_uuid: "66dd377b30ea48b8a33c9892e715e9f0" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 38711 } } peers { permanent_uuid: "f9b291a1bfb24cb0af222df109aac8ce" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 37639 } }
I20260806 02:58:40.666265 1279 raft_consensus.cc:374] T bb728e35dcd047adaaeda5b645ded748 P c5b5c4a10bc14bf6b7efe6cbdcfad492 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:58:40.666432 1279 raft_consensus.cc:729] T bb728e35dcd047adaaeda5b645ded748 P c5b5c4a10bc14bf6b7efe6cbdcfad492 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5b5c4a10bc14bf6b7efe6cbdcfad492, State: Initialized, Role: FOLLOWER
I20260806 02:58:40.666859 1279 consensus_queue.cc:260] T bb728e35dcd047adaaeda5b645ded748 P c5b5c4a10bc14bf6b7efe6cbdcfad492 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b5c4a10bc14bf6b7efe6cbdcfad492" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 36295 } } peers { permanent_uuid: "66dd377b30ea48b8a33c9892e715e9f0" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 38711 } } peers { permanent_uuid: "f9b291a1bfb24cb0af222df109aac8ce" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 37639 } }
I20260806 02:58:40.667425 1255 tablet_replica.cc:439] T bb728e35dcd047adaaeda5b645ded748 P c5b5c4a10bc14bf6b7efe6cbdcfad492: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b5c4a10bc14bf6b7efe6cbdcfad492" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 36295 } } peers { permanent_uuid: "66dd377b30ea48b8a33c9892e715e9f0" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 38711 } } peers { permanent_uuid: "f9b291a1bfb24cb0af222df109aac8ce" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 37639 } } }
I20260806 02:58:40.667600 1279 ts_tablet_manager.cc:1428] T bb728e35dcd047adaaeda5b645ded748 P c5b5c4a10bc14bf6b7efe6cbdcfad492: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260806 02:58:40.667765 1255 tablet_replica.cc:442] T bb728e35dcd047adaaeda5b645ded748 P c5b5c4a10bc14bf6b7efe6cbdcfad492: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:58:40.665823 1282 raft_consensus.cc:348] T bb728e35dcd047adaaeda5b645ded748 P f9b291a1bfb24cb0af222df109aac8ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b5c4a10bc14bf6b7efe6cbdcfad492" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 36295 } } peers { permanent_uuid: "66dd377b30ea48b8a33c9892e715e9f0" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 38711 } } peers { permanent_uuid: "f9b291a1bfb24cb0af222df109aac8ce" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 37639 } }
I20260806 02:58:40.669040 1282 raft_consensus.cc:374] T bb728e35dcd047adaaeda5b645ded748 P f9b291a1bfb24cb0af222df109aac8ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:58:40.669203 1282 raft_consensus.cc:729] T bb728e35dcd047adaaeda5b645ded748 P f9b291a1bfb24cb0af222df109aac8ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9b291a1bfb24cb0af222df109aac8ce, State: Initialized, Role: FOLLOWER
I20260806 02:58:40.669571 1282 consensus_queue.cc:260] T bb728e35dcd047adaaeda5b645ded748 P f9b291a1bfb24cb0af222df109aac8ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b5c4a10bc14bf6b7efe6cbdcfad492" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 36295 } } peers { permanent_uuid: "66dd377b30ea48b8a33c9892e715e9f0" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 38711 } } peers { permanent_uuid: "f9b291a1bfb24cb0af222df109aac8ce" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 37639 } }
I20260806 02:58:40.670190 1271 tablet_replica.cc:439] T bb728e35dcd047adaaeda5b645ded748 P f9b291a1bfb24cb0af222df109aac8ce: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b5c4a10bc14bf6b7efe6cbdcfad492" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 36295 } } peers { permanent_uuid: "66dd377b30ea48b8a33c9892e715e9f0" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 38711 } } peers { permanent_uuid: "f9b291a1bfb24cb0af222df109aac8ce" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 37639 } } }
I20260806 02:58:40.670560 1271 tablet_replica.cc:442] T bb728e35dcd047adaaeda5b645ded748 P f9b291a1bfb24cb0af222df109aac8ce: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:58:40.671648 1282 ts_tablet_manager.cc:1428] T bb728e35dcd047adaaeda5b645ded748 P f9b291a1bfb24cb0af222df109aac8ce: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20260806 02:58:40.673496 1278 tablet_bootstrap.cc:492] T bb728e35dcd047adaaeda5b645ded748 P 66dd377b30ea48b8a33c9892e715e9f0: No bootstrap required, opened a new log
I20260806 02:58:40.673799 1278 ts_tablet_manager.cc:1397] T bb728e35dcd047adaaeda5b645ded748 P 66dd377b30ea48b8a33c9892e715e9f0: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.000s
I20260806 02:58:40.674693 1278 raft_consensus.cc:348] T bb728e35dcd047adaaeda5b645ded748 P 66dd377b30ea48b8a33c9892e715e9f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b5c4a10bc14bf6b7efe6cbdcfad492" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 36295 } } peers { permanent_uuid: "66dd377b30ea48b8a33c9892e715e9f0" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 38711 } } peers { permanent_uuid: "f9b291a1bfb24cb0af222df109aac8ce" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 37639 } }
I20260806 02:58:40.675045 1278 raft_consensus.cc:374] T bb728e35dcd047adaaeda5b645ded748 P 66dd377b30ea48b8a33c9892e715e9f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:58:40.675136 1278 raft_consensus.cc:729] T bb728e35dcd047adaaeda5b645ded748 P 66dd377b30ea48b8a33c9892e715e9f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66dd377b30ea48b8a33c9892e715e9f0, State: Initialized, Role: FOLLOWER
I20260806 02:58:40.675439 1278 consensus_queue.cc:260] T bb728e35dcd047adaaeda5b645ded748 P 66dd377b30ea48b8a33c9892e715e9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b5c4a10bc14bf6b7efe6cbdcfad492" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 36295 } } peers { permanent_uuid: "66dd377b30ea48b8a33c9892e715e9f0" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 38711 } } peers { permanent_uuid: "f9b291a1bfb24cb0af222df109aac8ce" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 37639 } }
I20260806 02:58:40.675994 1256 tablet_replica.cc:439] T bb728e35dcd047adaaeda5b645ded748 P 66dd377b30ea48b8a33c9892e715e9f0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b5c4a10bc14bf6b7efe6cbdcfad492" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 36295 } } peers { permanent_uuid: "66dd377b30ea48b8a33c9892e715e9f0" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 38711 } } peers { permanent_uuid: "f9b291a1bfb24cb0af222df109aac8ce" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 37639 } } }
I20260806 02:58:40.676369 1256 tablet_replica.cc:442] T bb728e35dcd047adaaeda5b645ded748 P 66dd377b30ea48b8a33c9892e715e9f0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:58:40.677855 1278 ts_tablet_manager.cc:1428] T bb728e35dcd047adaaeda5b645ded748 P 66dd377b30ea48b8a33c9892e715e9f0: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260806 02:58:40.745496 1256 raft_consensus.cc:482] T bb728e35dcd047adaaeda5b645ded748 P 66dd377b30ea48b8a33c9892e715e9f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:58:40.745775 1256 raft_consensus.cc:504] T bb728e35dcd047adaaeda5b645ded748 P 66dd377b30ea48b8a33c9892e715e9f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b5c4a10bc14bf6b7efe6cbdcfad492" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 36295 } } peers { permanent_uuid: "66dd377b30ea48b8a33c9892e715e9f0" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 38711 } } peers { permanent_uuid: "f9b291a1bfb24cb0af222df109aac8ce" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 37639 } }
I20260806 02:58:40.747910 1256 leader_election.cc:288] T bb728e35dcd047adaaeda5b645ded748 P 66dd377b30ea48b8a33c9892e715e9f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5b5c4a10bc14bf6b7efe6cbdcfad492 (127.31.74.193:36295), f9b291a1bfb24cb0af222df109aac8ce (127.31.74.195:37639)
I20260806 02:58:40.758334 1247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:40.749026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.194:47797 (local address 127.31.74.195:37639)
0806 02:58:40.749245 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:40.749251 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:40.750400 (+ 1149us) server_negotiation.cc:407] Connection header received
0806 02:58:40.750568 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:40.750573 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:40.750692 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:40.750796 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:58:40.751494 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:40.752809 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:40.755419 (+ 2610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:40.756617 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:40.756989 (+ 372us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:40.757106 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:40.757560 (+ 454us) server_negotiation.cc:299] Negotiation successful
0806 02:58:40.757798 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260806 02:58:40.758347 1244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:40.748651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.194:36613 (local address 127.31.74.193:36295)
0806 02:58:40.748857 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:40.748866 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:40.749961 (+ 1095us) server_negotiation.cc:407] Connection header received
0806 02:58:40.750354 (+ 393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:40.750363 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:40.750453 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:40.750547 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:58:40.751342 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:40.752140 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:40.755029 (+ 2889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:40.756643 (+ 1614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:40.756980 (+ 337us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:40.757099 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:40.757722 (+ 623us) server_negotiation.cc:299] Negotiation successful
0806 02:58:40.757936 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260806 02:58:40.758435 1255 raft_consensus.cc:482] T bb728e35dcd047adaaeda5b645ded748 P c5b5c4a10bc14bf6b7efe6cbdcfad492 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:58:40.758724 1255 raft_consensus.cc:504] T bb728e35dcd047adaaeda5b645ded748 P c5b5c4a10bc14bf6b7efe6cbdcfad492 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b5c4a10bc14bf6b7efe6cbdcfad492" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 36295 } } peers { permanent_uuid: "66dd377b30ea48b8a33c9892e715e9f0" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 38711 } } peers { permanent_uuid: "f9b291a1bfb24cb0af222df109aac8ce" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 37639 } }
I20260806 02:58:40.759595 1300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:40.748435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.195:37639 (local address 127.31.74.194:47797)
0806 02:58:40.750274 (+ 1839us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:40.750299 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 02:58:40.750428 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:58:40.750809 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:58:40.750818 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:58:40.750838 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:58:40.751323 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:40.751336 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:40.752951 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:40.752958 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:40.755261 (+ 2303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:40.755272 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:40.756931 (+ 1659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:40.756939 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:40.757352 (+ 413us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:40.757387 (+ 35us) client_negotiation.cc:769] Sending connection context
0806 02:58:40.757495 (+ 108us) client_negotiation.cc:240] Negotiation successful
0806 02:58:40.757661 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1698,"mutex_wait_us":50,"thread_start_us":1480,"threads_started":1}
I20260806 02:58:40.759718 1255 leader_election.cc:288] T bb728e35dcd047adaaeda5b645ded748 P c5b5c4a10bc14bf6b7efe6cbdcfad492 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66dd377b30ea48b8a33c9892e715e9f0 (127.31.74.194:38711), f9b291a1bfb24cb0af222df109aac8ce (127.31.74.195:37639)
I20260806 02:58:40.759614 1299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:40.748311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:36295 (local address 127.31.74.194:36613)
0806 02:58:40.749825 (+ 1514us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:40.749846 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 02:58:40.750156 (+ 310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:58:40.750604 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:58:40.750608 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:58:40.750632 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:58:40.751056 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:40.751071 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:40.752287 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:40.752293 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:40.754790 (+ 2497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:40.754842 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:40.757370 (+ 2528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:40.757376 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:40.757544 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:40.757573 (+ 29us) client_negotiation.cc:769] Sending connection context
0806 02:58:40.758637 (+ 1064us) client_negotiation.cc:240] Negotiation successful
0806 02:58:40.758830 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1359,"mutex_wait_us":22,"thread_start_us":1150,"threads_started":1}
I20260806 02:58:40.760560 1035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb728e35dcd047adaaeda5b645ded748" candidate_uuid: "c5b5c4a10bc14bf6b7efe6cbdcfad492" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66dd377b30ea48b8a33c9892e715e9f0" is_pre_election: true
I20260806 02:58:40.760911 1035 raft_consensus.cc:2454] T bb728e35dcd047adaaeda5b645ded748 P 66dd377b30ea48b8a33c9892e715e9f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5b5c4a10bc14bf6b7efe6cbdcfad492 in term 0.
I20260806 02:58:40.760870 1179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb728e35dcd047adaaeda5b645ded748" candidate_uuid: "c5b5c4a10bc14bf6b7efe6cbdcfad492" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9b291a1bfb24cb0af222df109aac8ce" is_pre_election: true
I20260806 02:58:40.761786 817 leader_election.cc:302] T bb728e35dcd047adaaeda5b645ded748 P c5b5c4a10bc14bf6b7efe6cbdcfad492 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66dd377b30ea48b8a33c9892e715e9f0, c5b5c4a10bc14bf6b7efe6cbdcfad492; no voters:
I20260806 02:58:40.761786 889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb728e35dcd047adaaeda5b645ded748" candidate_uuid: "66dd377b30ea48b8a33c9892e715e9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5b5c4a10bc14bf6b7efe6cbdcfad492" is_pre_election: true
I20260806 02:58:40.762133 889 raft_consensus.cc:2454] T bb728e35dcd047adaaeda5b645ded748 P c5b5c4a10bc14bf6b7efe6cbdcfad492 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66dd377b30ea48b8a33c9892e715e9f0 in term 0.
I20260806 02:58:40.762146 1179 raft_consensus.cc:2454] T bb728e35dcd047adaaeda5b645ded748 P f9b291a1bfb24cb0af222df109aac8ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5b5c4a10bc14bf6b7efe6cbdcfad492 in term 0.
I20260806 02:58:40.762362 1255 raft_consensus.cc:2781] T bb728e35dcd047adaaeda5b645ded748 P c5b5c4a10bc14bf6b7efe6cbdcfad492 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:58:40.762501 1255 raft_consensus.cc:482] T bb728e35dcd047adaaeda5b645ded748 P c5b5c4a10bc14bf6b7efe6cbdcfad492 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:58:40.762666 1255 raft_consensus.cc:3037] T bb728e35dcd047adaaeda5b645ded748 P c5b5c4a10bc14bf6b7efe6cbdcfad492 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:58:40.762902 965 leader_election.cc:302] T bb728e35dcd047adaaeda5b645ded748 P 66dd377b30ea48b8a33c9892e715e9f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66dd377b30ea48b8a33c9892e715e9f0, c5b5c4a10bc14bf6b7efe6cbdcfad492; no voters:
I20260806 02:58:40.763527 1256 raft_consensus.cc:2781] T bb728e35dcd047adaaeda5b645ded748 P 66dd377b30ea48b8a33c9892e715e9f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:58:40.762928 1178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb728e35dcd047adaaeda5b645ded748" candidate_uuid: "66dd377b30ea48b8a33c9892e715e9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9b291a1bfb24cb0af222df109aac8ce" is_pre_election: true
I20260806 02:58:40.763700 1256 raft_consensus.cc:482] T bb728e35dcd047adaaeda5b645ded748 P 66dd377b30ea48b8a33c9892e715e9f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:58:40.763998 1256 raft_consensus.cc:3037] T bb728e35dcd047adaaeda5b645ded748 P 66dd377b30ea48b8a33c9892e715e9f0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:58:40.763978 1178 raft_consensus.cc:2454] T bb728e35dcd047adaaeda5b645ded748 P f9b291a1bfb24cb0af222df109aac8ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66dd377b30ea48b8a33c9892e715e9f0 in term 0.
I20260806 02:58:40.768878 1256 raft_consensus.cc:504] T bb728e35dcd047adaaeda5b645ded748 P 66dd377b30ea48b8a33c9892e715e9f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b5c4a10bc14bf6b7efe6cbdcfad492" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 36295 } } peers { permanent_uuid: "66dd377b30ea48b8a33c9892e715e9f0" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 38711 } } peers { permanent_uuid: "f9b291a1bfb24cb0af222df109aac8ce" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 37639 } }
I20260806 02:58:40.769170 1255 raft_consensus.cc:504] T bb728e35dcd047adaaeda5b645ded748 P c5b5c4a10bc14bf6b7efe6cbdcfad492 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b5c4a10bc14bf6b7efe6cbdcfad492" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 36295 } } peers { permanent_uuid: "66dd377b30ea48b8a33c9892e715e9f0" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 38711 } } peers { permanent_uuid: "f9b291a1bfb24cb0af222df109aac8ce" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 37639 } }
I20260806 02:58:40.770788 1035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb728e35dcd047adaaeda5b645ded748" candidate_uuid: "c5b5c4a10bc14bf6b7efe6cbdcfad492" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66dd377b30ea48b8a33c9892e715e9f0"
I20260806 02:58:40.771395 1035 raft_consensus.cc:2379] T bb728e35dcd047adaaeda5b645ded748 P 66dd377b30ea48b8a33c9892e715e9f0 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate c5b5c4a10bc14bf6b7efe6cbdcfad492 in current term 1: Already voted for candidate 66dd377b30ea48b8a33c9892e715e9f0 in this term.
I20260806 02:58:40.771302 889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb728e35dcd047adaaeda5b645ded748" candidate_uuid: "66dd377b30ea48b8a33c9892e715e9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5b5c4a10bc14bf6b7efe6cbdcfad492"
I20260806 02:58:40.771505 1178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb728e35dcd047adaaeda5b645ded748" candidate_uuid: "66dd377b30ea48b8a33c9892e715e9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9b291a1bfb24cb0af222df109aac8ce"
I20260806 02:58:40.771936 1178 raft_consensus.cc:3037] T bb728e35dcd047adaaeda5b645ded748 P f9b291a1bfb24cb0af222df109aac8ce [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:58:40.772085 889 raft_consensus.cc:2379] T bb728e35dcd047adaaeda5b645ded748 P c5b5c4a10bc14bf6b7efe6cbdcfad492 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 66dd377b30ea48b8a33c9892e715e9f0 in current term 1: Already voted for candidate c5b5c4a10bc14bf6b7efe6cbdcfad492 in this term.
I20260806 02:58:40.772508 1256 leader_election.cc:288] T bb728e35dcd047adaaeda5b645ded748 P 66dd377b30ea48b8a33c9892e715e9f0 [CANDIDATE]: Term 1 election: Requested vote from peers c5b5c4a10bc14bf6b7efe6cbdcfad492 (127.31.74.193:36295), f9b291a1bfb24cb0af222df109aac8ce (127.31.74.195:37639)
I20260806 02:58:40.772939 1255 leader_election.cc:288] T bb728e35dcd047adaaeda5b645ded748 P c5b5c4a10bc14bf6b7efe6cbdcfad492 [CANDIDATE]: Term 1 election: Requested vote from peers 66dd377b30ea48b8a33c9892e715e9f0 (127.31.74.194:38711), f9b291a1bfb24cb0af222df109aac8ce (127.31.74.195:37639)
I20260806 02:58:40.773833 1179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb728e35dcd047adaaeda5b645ded748" candidate_uuid: "c5b5c4a10bc14bf6b7efe6cbdcfad492" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9b291a1bfb24cb0af222df109aac8ce"
I20260806 02:58:40.774691 817 leader_election.cc:302] T bb728e35dcd047adaaeda5b645ded748 P c5b5c4a10bc14bf6b7efe6cbdcfad492 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5b5c4a10bc14bf6b7efe6cbdcfad492; no voters: 66dd377b30ea48b8a33c9892e715e9f0, f9b291a1bfb24cb0af222df109aac8ce
I20260806 02:58:40.775068 1255 raft_consensus.cc:2726] T bb728e35dcd047adaaeda5b645ded748 P c5b5c4a10bc14bf6b7efe6cbdcfad492 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260806 02:58:40.777420 1178 raft_consensus.cc:2454] T bb728e35dcd047adaaeda5b645ded748 P f9b291a1bfb24cb0af222df109aac8ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66dd377b30ea48b8a33c9892e715e9f0 in term 1.
I20260806 02:58:40.778321 964 leader_election.cc:302] T bb728e35dcd047adaaeda5b645ded748 P 66dd377b30ea48b8a33c9892e715e9f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66dd377b30ea48b8a33c9892e715e9f0, f9b291a1bfb24cb0af222df109aac8ce; no voters: c5b5c4a10bc14bf6b7efe6cbdcfad492
I20260806 02:58:40.779311 1256 raft_consensus.cc:2781] T bb728e35dcd047adaaeda5b645ded748 P 66dd377b30ea48b8a33c9892e715e9f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:58:40.780272 1256 raft_consensus.cc:686] T bb728e35dcd047adaaeda5b645ded748 P 66dd377b30ea48b8a33c9892e715e9f0 [term 1 LEADER]: Becoming Leader. State: Replica: 66dd377b30ea48b8a33c9892e715e9f0, State: Running, Role: LEADER
I20260806 02:58:40.781586 1256 consensus_queue.cc:237] T bb728e35dcd047adaaeda5b645ded748 P 66dd377b30ea48b8a33c9892e715e9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b5c4a10bc14bf6b7efe6cbdcfad492" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 36295 } } peers { permanent_uuid: "66dd377b30ea48b8a33c9892e715e9f0" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 38711 } } peers { permanent_uuid: "f9b291a1bfb24cb0af222df109aac8ce" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 37639 } }
I20260806 02:58:40.786479 1301 tablet_replica.cc:439] T bb728e35dcd047adaaeda5b645ded748 P 66dd377b30ea48b8a33c9892e715e9f0: TxnStatusTablet state changed. Reason: New leader 66dd377b30ea48b8a33c9892e715e9f0. Latest consensus state: current_term: 1 leader_uuid: "66dd377b30ea48b8a33c9892e715e9f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b5c4a10bc14bf6b7efe6cbdcfad492" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 36295 } } peers { permanent_uuid: "66dd377b30ea48b8a33c9892e715e9f0" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 38711 } } peers { permanent_uuid: "f9b291a1bfb24cb0af222df109aac8ce" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 37639 } } }
I20260806 02:58:40.787730 1301 tablet_replica.cc:442] T bb728e35dcd047adaaeda5b645ded748 P 66dd377b30ea48b8a33c9892e715e9f0: This TxnStatusTablet replica's current role is: LEADER
I20260806 02:58:40.789098 726 catalog_manager.cc:5495] T bb728e35dcd047adaaeda5b645ded748 P 66dd377b30ea48b8a33c9892e715e9f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66dd377b30ea48b8a33c9892e715e9f0 (127.31.74.194). New cstate: current_term: 1 leader_uuid: "66dd377b30ea48b8a33c9892e715e9f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b5c4a10bc14bf6b7efe6cbdcfad492" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 36295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66dd377b30ea48b8a33c9892e715e9f0" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 38711 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9b291a1bfb24cb0af222df109aac8ce" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 37639 } health_report { overall_health: UNKNOWN } } }
I20260806 02:58:40.804447 1303 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260806 02:58:40.805184 1303 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260806 02:58:40.845441 1295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:40.836494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.194:38711 (local address 127.0.0.1:40840)
0806 02:58:40.836887 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:40.836905 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:58:40.837092 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:58:40.837635 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:58:40.837640 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:58:40.837653 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 02:58:40.837862 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:40.837871 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:40.839736 (+ 1865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:40.839742 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:40.841471 (+ 1729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:40.841489 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:40.843059 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:40.843066 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:40.843331 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:40.843422 (+ 91us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 02:58:40.844443 (+ 1021us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 02:58:40.844530 (+ 87us) client_negotiation.cc:769] Sending connection context
0806 02:58:40.844651 (+ 121us) client_negotiation.cc:240] Negotiation successful
0806 02:58:40.844890 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171}
I20260806 02:58:40.845507 1246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:40.836421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40840 (local address 127.31.74.194:38711)
0806 02:58:40.836636 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:40.836643 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:40.836995 (+ 352us) server_negotiation.cc:407] Connection header received
0806 02:58:40.837308 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:40.837313 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:40.837459 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:40.837610 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:58:40.838002 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:40.839548 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:40.841676 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:40.842574 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:40.843028 (+ 454us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:40.843710 (+ 682us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:58:40.844318 (+ 608us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:58:40.844481 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:40.844709 (+ 228us) server_negotiation.cc:299] Negotiation successful
0806 02:58:40.844812 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260806 02:58:40.853775 889 raft_consensus.cc:1261] T bb728e35dcd047adaaeda5b645ded748 P c5b5c4a10bc14bf6b7efe6cbdcfad492 [term 1 FOLLOWER]: Refusing update from remote peer 66dd377b30ea48b8a33c9892e715e9f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:58:40.854103 1178 raft_consensus.cc:1261] T bb728e35dcd047adaaeda5b645ded748 P f9b291a1bfb24cb0af222df109aac8ce [term 1 FOLLOWER]: Refusing update from remote peer 66dd377b30ea48b8a33c9892e715e9f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:58:40.855270 1301 consensus_queue.cc:1035] T bb728e35dcd047adaaeda5b645ded748 P 66dd377b30ea48b8a33c9892e715e9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5b5c4a10bc14bf6b7efe6cbdcfad492" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 36295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:58:40.856114 1256 consensus_queue.cc:1035] T bb728e35dcd047adaaeda5b645ded748 P 66dd377b30ea48b8a33c9892e715e9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9b291a1bfb24cb0af222df109aac8ce" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 37639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:58:40.872413 1255 tablet_replica.cc:439] T bb728e35dcd047adaaeda5b645ded748 P c5b5c4a10bc14bf6b7efe6cbdcfad492: TxnStatusTablet state changed. Reason: New leader 66dd377b30ea48b8a33c9892e715e9f0. Latest consensus state: current_term: 1 leader_uuid: "66dd377b30ea48b8a33c9892e715e9f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b5c4a10bc14bf6b7efe6cbdcfad492" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 36295 } } peers { permanent_uuid: "66dd377b30ea48b8a33c9892e715e9f0" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 38711 } } peers { permanent_uuid: "f9b291a1bfb24cb0af222df109aac8ce" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 37639 } } }
I20260806 02:58:40.872936 1255 tablet_replica.cc:442] T bb728e35dcd047adaaeda5b645ded748 P c5b5c4a10bc14bf6b7efe6cbdcfad492: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:58:40.875550 1301 tablet_replica.cc:439] T bb728e35dcd047adaaeda5b645ded748 P 66dd377b30ea48b8a33c9892e715e9f0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "66dd377b30ea48b8a33c9892e715e9f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b5c4a10bc14bf6b7efe6cbdcfad492" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 36295 } } peers { permanent_uuid: "66dd377b30ea48b8a33c9892e715e9f0" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 38711 } } peers { permanent_uuid: "f9b291a1bfb24cb0af222df109aac8ce" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 37639 } } }
I20260806 02:58:40.875989 1301 tablet_replica.cc:442] T bb728e35dcd047adaaeda5b645ded748 P 66dd377b30ea48b8a33c9892e715e9f0: This TxnStatusTablet replica's current role is: LEADER
I20260806 02:58:40.877314 1265 mvcc.cc:205] Tried to move back new op lower bound from 7315395055005847552 to 7315395054727573504. Current Snapshot: MvccSnapshot[applied={T|T < 7315395055005847552}]
I20260806 02:58:40.878208 1255 tablet_replica.cc:439] T bb728e35dcd047adaaeda5b645ded748 P c5b5c4a10bc14bf6b7efe6cbdcfad492: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "66dd377b30ea48b8a33c9892e715e9f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b5c4a10bc14bf6b7efe6cbdcfad492" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 36295 } } peers { permanent_uuid: "66dd377b30ea48b8a33c9892e715e9f0" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 38711 } } peers { permanent_uuid: "f9b291a1bfb24cb0af222df109aac8ce" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 37639 } } }
I20260806 02:58:40.878825 1255 tablet_replica.cc:442] T bb728e35dcd047adaaeda5b645ded748 P c5b5c4a10bc14bf6b7efe6cbdcfad492: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:58:40.886433 1271 tablet_replica.cc:439] T bb728e35dcd047adaaeda5b645ded748 P f9b291a1bfb24cb0af222df109aac8ce: TxnStatusTablet state changed. Reason: New leader 66dd377b30ea48b8a33c9892e715e9f0. Latest consensus state: current_term: 1 leader_uuid: "66dd377b30ea48b8a33c9892e715e9f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b5c4a10bc14bf6b7efe6cbdcfad492" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 36295 } } peers { permanent_uuid: "66dd377b30ea48b8a33c9892e715e9f0" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 38711 } } peers { permanent_uuid: "f9b291a1bfb24cb0af222df109aac8ce" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 37639 } } }
I20260806 02:58:40.887244 1271 tablet_replica.cc:442] T bb728e35dcd047adaaeda5b645ded748 P f9b291a1bfb24cb0af222df109aac8ce: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:58:40.892336 1256 tablet_replica.cc:439] T bb728e35dcd047adaaeda5b645ded748 P 66dd377b30ea48b8a33c9892e715e9f0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "66dd377b30ea48b8a33c9892e715e9f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b5c4a10bc14bf6b7efe6cbdcfad492" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 36295 } } peers { permanent_uuid: "66dd377b30ea48b8a33c9892e715e9f0" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 38711 } } peers { permanent_uuid: "f9b291a1bfb24cb0af222df109aac8ce" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 37639 } } }
I20260806 02:58:40.892907 1256 tablet_replica.cc:442] T bb728e35dcd047adaaeda5b645ded748 P 66dd377b30ea48b8a33c9892e715e9f0: This TxnStatusTablet replica's current role is: LEADER
I20260806 02:58:40.895392 1271 tablet_replica.cc:439] T bb728e35dcd047adaaeda5b645ded748 P f9b291a1bfb24cb0af222df109aac8ce: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "66dd377b30ea48b8a33c9892e715e9f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5b5c4a10bc14bf6b7efe6cbdcfad492" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 36295 } } peers { permanent_uuid: "66dd377b30ea48b8a33c9892e715e9f0" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 38711 } } peers { permanent_uuid: "f9b291a1bfb24cb0af222df109aac8ce" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 37639 } } }
I20260806 02:58:40.895764 1271 tablet_replica.cc:442] T bb728e35dcd047adaaeda5b645ded748 P f9b291a1bfb24cb0af222df109aac8ce: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:58:40.910215 1244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:40.892088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50004 (local address 127.31.74.193:36295)
0806 02:58:40.893212 (+ 1124us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:40.893238 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:40.900851 (+ 7613us) server_negotiation.cc:407] Connection header received
0806 02:58:40.901784 (+ 933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:40.901796 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:40.902002 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:40.902229 (+ 227us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:58:40.902850 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:40.903953 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:40.905598 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:40.907501 (+ 1903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:40.907897 (+ 396us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:40.908451 (+ 554us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:58:40.909097 (+ 646us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:58:40.909277 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:40.909403 (+ 126us) server_negotiation.cc:299] Negotiation successful
0806 02:58:40.909512 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":957}
I20260806 02:58:40.942334 1246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:40.931501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.193:39275 (local address 127.31.74.194:38711)
0806 02:58:40.931967 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:40.931987 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:40.934740 (+ 2753us) server_negotiation.cc:407] Connection header received
0806 02:58:40.935025 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:40.935036 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:40.935116 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:40.935220 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:58:40.935998 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:40.937369 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:40.939369 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:40.940381 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:40.940542 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:40.941228 (+ 686us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:58:40.941382 (+ 154us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:58:40.941522 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:40.941693 (+ 171us) server_negotiation.cc:299] Negotiation successful
0806 02:58:40.941799 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153}
I20260806 02:58:40.942523 1312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:40.932006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.194:38711 (local address 127.31.74.193:39275)
0806 02:58:40.934547 (+ 2541us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:40.934569 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 02:58:40.934868 (+ 299us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:58:40.935238 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:58:40.935242 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:58:40.935254 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 02:58:40.935814 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:40.935826 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:40.937536 (+ 1710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:40.937542 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:40.939182 (+ 1640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:40.939198 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:40.940629 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:40.940635 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:40.940830 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:40.940902 (+ 72us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 02:58:40.941525 (+ 623us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 02:58:40.941542 (+ 17us) client_negotiation.cc:769] Sending connection context
0806 02:58:40.941649 (+ 107us) client_negotiation.cc:240] Negotiation successful
0806 02:58:40.941825 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2248,"thread_start_us":1800,"threads_started":1}
I20260806 02:58:40.957969 1035 raft_consensus.cc:1261] T e7f0f8fdbb384284931e197973452a79 P 66dd377b30ea48b8a33c9892e715e9f0 [term 1 FOLLOWER]: Refusing update from remote peer c5b5c4a10bc14bf6b7efe6cbdcfad492: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:58:40.958083 1178 raft_consensus.cc:1261] T e7f0f8fdbb384284931e197973452a79 P f9b291a1bfb24cb0af222df109aac8ce [term 1 FOLLOWER]: Refusing update from remote peer c5b5c4a10bc14bf6b7efe6cbdcfad492: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:58:40.958853 1286 consensus_queue.cc:1035] T e7f0f8fdbb384284931e197973452a79 P c5b5c4a10bc14bf6b7efe6cbdcfad492 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9b291a1bfb24cb0af222df109aac8ce" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 37639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:58:40.959443 1255 consensus_queue.cc:1035] T e7f0f8fdbb384284931e197973452a79 P c5b5c4a10bc14bf6b7efe6cbdcfad492 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66dd377b30ea48b8a33c9892e715e9f0" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 38711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:58:40.982141 1266 mvcc.cc:205] Tried to move back new op lower bound from 7315395055435911168 to 7315395053754265600. Current Snapshot: MvccSnapshot[applied={T|T < 7315395055435911168}]
I20260806 02:58:41.029733 1244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:41.017376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.194:40889 (local address 127.31.74.193:36295)
0806 02:58:41.017627 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:41.017636 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:41.020278 (+ 2642us) server_negotiation.cc:407] Connection header received
0806 02:58:41.020523 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:41.020529 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:41.020632 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:41.020749 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:58:41.022307 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:41.023987 (+ 1680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:41.026683 (+ 2696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:41.027578 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:41.027732 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:41.028563 (+ 831us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:58:41.028728 (+ 165us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:58:41.028819 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:41.029091 (+ 272us) server_negotiation.cc:299] Negotiation successful
0806 02:58:41.029197 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260806 02:58:41.030925 1316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:41.017035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:36295 (local address 127.31.74.194:40889)
0806 02:58:41.020143 (+ 3108us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:41.020166 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 02:58:41.020341 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:58:41.020800 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:58:41.020804 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:58:41.020822 (+ 18us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 02:58:41.021550 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:41.021566 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:41.024377 (+ 2811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:41.024383 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:41.026437 (+ 2054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:41.026456 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:41.027964 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:41.027975 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:41.028201 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:41.028308 (+ 107us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 02:58:41.028925 (+ 617us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 02:58:41.028938 (+ 13us) client_negotiation.cc:769] Sending connection context
0806 02:58:41.030113 (+ 1175us) client_negotiation.cc:240] Negotiation successful
0806 02:58:41.030325 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2851,"thread_start_us":2648,"threads_started":1}
I20260806 02:58:41.115376 726 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:58:41.115685 726 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:58:41.120030 726 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b5e6b82242fc46519f1bfc172a68c982: Sending DeleteTablet for 3 replicas of tablet e7f0f8fdbb384284931e197973452a79
I20260806 02:58:41.121769 1159 tablet_service.cc:1514] Processing DeleteTablet for tablet e7f0f8fdbb384284931e197973452a79 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:58:41 UTC) from {username='kudu', principal='kudu/127.31.74.254@KRBTEST.COM'} at 127.0.0.1:43874
I20260806 02:58:41.121883 1015 tablet_service.cc:1514] Processing DeleteTablet for tablet e7f0f8fdbb384284931e197973452a79 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:58:41 UTC) from {username='kudu', principal='kudu/127.31.74.254@KRBTEST.COM'} at 127.0.0.1:40830
I20260806 02:58:41.123495 867 tablet_service.cc:1514] Processing DeleteTablet for tablet e7f0f8fdbb384284931e197973452a79 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:58:41 UTC) from {username='kudu', principal='kudu/127.31.74.254@KRBTEST.COM'} at 127.0.0.1:49994
I20260806 02:58:41.143409 1320 tablet_replica.cc:330] T e7f0f8fdbb384284931e197973452a79 P f9b291a1bfb24cb0af222df109aac8ce: stopping tablet replica
I20260806 02:58:41.143922 1320 raft_consensus.cc:2229] T e7f0f8fdbb384284931e197973452a79 P f9b291a1bfb24cb0af222df109aac8ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:58:41.144281 1320 raft_consensus.cc:2258] T e7f0f8fdbb384284931e197973452a79 P f9b291a1bfb24cb0af222df109aac8ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:58:41.145830 1320 ts_tablet_manager.cc:1905] T e7f0f8fdbb384284931e197973452a79 P f9b291a1bfb24cb0af222df109aac8ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:58:41.155488 1321 tablet_replica.cc:330] T e7f0f8fdbb384284931e197973452a79 P c5b5c4a10bc14bf6b7efe6cbdcfad492: stopping tablet replica
I20260806 02:58:41.156225 1319 tablet_replica.cc:330] T e7f0f8fdbb384284931e197973452a79 P 66dd377b30ea48b8a33c9892e715e9f0: stopping tablet replica
I20260806 02:58:41.156585 1321 raft_consensus.cc:2229] T e7f0f8fdbb384284931e197973452a79 P c5b5c4a10bc14bf6b7efe6cbdcfad492 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:58:41.157186 1319 raft_consensus.cc:2229] T e7f0f8fdbb384284931e197973452a79 P 66dd377b30ea48b8a33c9892e715e9f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:58:41.157789 1321 raft_consensus.cc:2258] T e7f0f8fdbb384284931e197973452a79 P c5b5c4a10bc14bf6b7efe6cbdcfad492 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:58:41.158324 1319 raft_consensus.cc:2258] T e7f0f8fdbb384284931e197973452a79 P 66dd377b30ea48b8a33c9892e715e9f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:58:41.159435 1321 ts_tablet_manager.cc:1905] T e7f0f8fdbb384284931e197973452a79 P c5b5c4a10bc14bf6b7efe6cbdcfad492: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:58:41.165556 1319 ts_tablet_manager.cc:1905] T e7f0f8fdbb384284931e197973452a79 P 66dd377b30ea48b8a33c9892e715e9f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:58:41.170725 1320 ts_tablet_manager.cc:1918] T e7f0f8fdbb384284931e197973452a79 P f9b291a1bfb24cb0af222df109aac8ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:58:41.171016 1320 log.cc:1196] T e7f0f8fdbb384284931e197973452a79 P f9b291a1bfb24cb0af222df109aac8ce: Deleting WAL directory at /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/minicluster-data/ts-2/wal/wals/e7f0f8fdbb384284931e197973452a79
I20260806 02:58:41.173179 1320 ts_tablet_manager.cc:1939] T e7f0f8fdbb384284931e197973452a79 P f9b291a1bfb24cb0af222df109aac8ce: Deleting consensus metadata
I20260806 02:58:41.179360 1321 ts_tablet_manager.cc:1918] T e7f0f8fdbb384284931e197973452a79 P c5b5c4a10bc14bf6b7efe6cbdcfad492: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:58:41.179441 1319 ts_tablet_manager.cc:1918] T e7f0f8fdbb384284931e197973452a79 P 66dd377b30ea48b8a33c9892e715e9f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:58:41.181162 710 catalog_manager.cc:4841] TS f9b291a1bfb24cb0af222df109aac8ce (127.31.74.195:37639): tablet e7f0f8fdbb384284931e197973452a79 (table test-table [id=319d99217062406c9cee71c4b0ded160]) successfully deleted
I20260806 02:58:41.181560 1319 log.cc:1196] T e7f0f8fdbb384284931e197973452a79 P 66dd377b30ea48b8a33c9892e715e9f0: Deleting WAL directory at /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/minicluster-data/ts-1/wal/wals/e7f0f8fdbb384284931e197973452a79
I20260806 02:58:41.182327 1319 ts_tablet_manager.cc:1939] T e7f0f8fdbb384284931e197973452a79 P 66dd377b30ea48b8a33c9892e715e9f0: Deleting consensus metadata
Aug 06 02:58:41 dist-test-slave-dhph krb5kdc[680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985119, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.74.193@KRBTEST.COM
I20260806 02:58:41.186331 1321 log.cc:1196] T e7f0f8fdbb384284931e197973452a79 P c5b5c4a10bc14bf6b7efe6cbdcfad492: Deleting WAL directory at /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785985105255475-32043-0/minicluster-data/ts-0/wal/wals/e7f0f8fdbb384284931e197973452a79
I20260806 02:58:41.187023 1321 ts_tablet_manager.cc:1939] T e7f0f8fdbb384284931e197973452a79 P c5b5c4a10bc14bf6b7efe6cbdcfad492: Deleting consensus metadata
I20260806 02:58:41.187825 710 catalog_manager.cc:4841] TS 66dd377b30ea48b8a33c9892e715e9f0 (127.31.74.194:38711): tablet e7f0f8fdbb384284931e197973452a79 (table test-table [id=319d99217062406c9cee71c4b0ded160]) successfully deleted
I20260806 02:58:41.188356 711 catalog_manager.cc:4841] TS c5b5c4a10bc14bf6b7efe6cbdcfad492 (127.31.74.193:36295): tablet e7f0f8fdbb384284931e197973452a79 (table test-table [id=319d99217062406c9cee71c4b0ded160]) successfully deleted
I20260806 02:58:41.199572 1244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:41.156254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50010 (local address 127.31.74.193:36295)
0806 02:58:41.156462 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:41.156469 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:41.164572 (+ 8103us) server_negotiation.cc:407] Connection header received
0806 02:58:41.167026 (+ 2454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:41.167033 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:41.167125 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:41.167210 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:41.173076 (+ 5866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:41.174499 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:41.179678 (+ 5179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:41.180481 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:41.180588 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:41.189978 (+ 9390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:58:41.190000 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:58:41.190022 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:58:41.190070 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:58:41.194838 (+ 4768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:41.196019 (+ 1181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:41.196030 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:41.196034 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:41.196223 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:41.196721 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:41.196727 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:41.196730 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:41.197372 (+ 642us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:58:41.197740 (+ 368us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:41.198715 (+ 975us) server_negotiation.cc:299] Negotiation successful
0806 02:58:41.199087 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260806 02:58:41.201979 938 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50010
I20260806 02:58:41.242004 1327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:41.216357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39820 (local address 127.31.74.254:37001)
0806 02:58:41.219126 (+ 2769us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:41.219135 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:41.219722 (+ 587us) server_negotiation.cc:407] Connection header received
0806 02:58:41.219956 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:41.219964 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:41.220064 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:41.220218 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:41.222404 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:41.224233 (+ 1829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:41.226701 (+ 2468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:41.228012 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:41.228574 (+ 562us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:41.232684 (+ 4110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:58:41.232693 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:58:41.232699 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:58:41.232763 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:58:41.237571 (+ 4808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:41.238645 (+ 1074us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:41.238652 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:41.238656 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:41.238857 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:41.239356 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:41.239362 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:41.239366 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:41.239848 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:58:41.240157 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:41.240636 (+ 479us) server_negotiation.cc:299] Negotiation successful
0806 02:58:41.241561 (+ 925us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2611,"thread_start_us":2442,"threads_started":1}
W20260806 02:58:41.242769 726 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39820
I20260806 02:58:41.246002 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 799
I20260806 02:58:41.277621 1244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:41.247417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50026 (local address 127.31.74.193:36295)
0806 02:58:41.248057 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:41.248068 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:41.249961 (+ 1893us) server_negotiation.cc:407] Connection header received
0806 02:58:41.250321 (+ 360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:41.250328 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:41.250422 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:41.250518 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:41.254012 (+ 3494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:41.256542 (+ 2530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:41.259478 (+ 2936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:41.260730 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:41.260862 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:41.265174 (+ 4312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:58:41.265183 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:58:41.265189 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:58:41.265257 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:58:41.272586 (+ 7329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:41.273764 (+ 1178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:41.273774 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:41.273778 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:41.273978 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:41.274514 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:41.274519 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:41.274523 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:41.274997 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:58:41.275406 (+ 409us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:41.276897 (+ 1491us) server_negotiation.cc:299] Negotiation successful
0806 02:58:41.277211 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280}
W20260806 02:58:41.278311 938 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50026
W20260806 02:58:41.279050 32043 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 799: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 02:58:41.279291 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 799
I20260806 02:58:41.303063 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 947
Aug 06 02:58:41 dist-test-slave-dhph krb5kdc[680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985119, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.74.194@KRBTEST.COM
I20260806 02:58:41.324026 1246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:41.303940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40848 (local address 127.31.74.194:38711)
0806 02:58:41.304142 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:41.304147 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:41.304234 (+ 87us) server_negotiation.cc:407] Connection header received
0806 02:58:41.304497 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:41.304502 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:41.304599 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:41.304691 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:41.306859 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:41.307665 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:41.309822 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:41.310367 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:41.310489 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:41.316655 (+ 6166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:58:41.316674 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:58:41.316691 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:58:41.316748 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:58:41.320621 (+ 3873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:41.321525 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:41.321534 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:41.321536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:41.321715 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:41.322153 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:41.322160 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:41.322164 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:41.322691 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:58:41.323063 (+ 372us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:41.323366 (+ 303us) server_negotiation.cc:299] Negotiation successful
0806 02:58:41.323632 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
W20260806 02:58:41.324777 1081 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40848
W20260806 02:58:41.325943 32043 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 947: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 02:58:41.326149 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 947
I20260806 02:58:41.355427 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 1090
Aug 06 02:58:41 dist-test-slave-dhph krb5kdc[680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985119, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.74.195@KRBTEST.COM
I20260806 02:58:41.376582 1329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:41.356376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43894 (local address 127.31.74.195:37639)
0806 02:58:41.357184 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:41.357191 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:41.357214 (+ 23us) server_negotiation.cc:407] Connection header received
0806 02:58:41.357285 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:41.357291 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:41.357423 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:41.357547 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:41.359663 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:41.360387 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:41.361744 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:41.362182 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:41.362327 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:41.367617 (+ 5290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:58:41.367635 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:58:41.367654 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:58:41.367708 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:58:41.371372 (+ 3664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:41.373182 (+ 1810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:41.373196 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:41.373199 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:41.373451 (+ 252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:41.374087 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:41.374092 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:41.374096 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:41.374683 (+ 587us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:58:41.375049 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:41.376050 (+ 1001us) server_negotiation.cc:299] Negotiation successful
0806 02:58:41.376249 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":358,"threads_started":1}
W20260806 02:58:41.377619 1225 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43894
W20260806 02:58:41.378873 32043 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 1090: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 02:58:41.379031 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 1090
I20260806 02:58:41.410387 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 693
I20260806 02:58:41.411916 768 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:58:41.545514 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 693
2026-08-06T02:58:41Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (6017 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:58:41 dist-test-slave-dhph krb5kdc[1333](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:58:41 dist-test-slave-dhph krb5kdc[1333](info): set up 2 sockets
Aug 06 02:58:41 dist-test-slave-dhph krb5kdc[1333](info): commencing operation
krb5kdc: starting...
W20260806 02:58:42.825186 32043 mini_kdc.cc:121] Time spent starting KDC: real 1.225s user 0.007s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/krb5kdc/test-user.keytab.
Aug 06 02:58:43 dist-test-slave-dhph krb5kdc[1333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:58:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:58:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.74.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab.
Entry for principal kudu/127.31.74.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab.
I20260806 02:58:43.125875 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:42053
--webserver_interface=127.31.74.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab
--principal=kudu/127.31.74.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:46815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.74.254:42053
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:58:43.575136 1346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:58:43.575459 1346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:58:43.575552 1346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:58:43.585300 1346 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:58:43.585477 1346 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260806 02:58:43.585562 1346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:58:43.585705 1346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:58:43.585790 1346 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:58:43.585980 1346 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:58:43.599162 1346 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:46815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.74.254:42053
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab
--principal=kudu/127.31.74.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:42053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.74.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:58:43.600636 1346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:58:43.602320 1346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:58:43.618031 1351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:58:43.618091 1354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:58:43.618317 1352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:58:43.619344 1346 server_base.cc:1034] running on GCE node
I20260806 02:58:43.620569 1346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:58:43.623143 1346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:58:43.624739 1346 hybrid_clock.cc:648] HybridClock initialized: now 1785985123624657 us; error 49 us; skew 500 ppm
I20260806 02:58:43.625128 1346 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:58:43 dist-test-slave-dhph krb5kdc[1333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985123, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:58:43.635357 1346 init.cc:377] Logged in from keytab as kudu/127.31.74.254@KRBTEST.COM (short username kudu)
I20260806 02:58:43.637468 1346 webserver.cc:460] Webserver started at http://127.31.74.254:35777/ using document root <none> and password file <none>
I20260806 02:58:43.638803 1346 fs_manager.cc:359] Metadata directory not provided
I20260806 02:58:43.638922 1346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:58:43.639398 1346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:58:43.643705 1346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/minicluster-data/master-0/data/instance:
uuid: "24ba9714cb1a494d983288e0ab1b4283"
format_stamp: "Formatted at 2026-08-06 02:58:43 on dist-test-slave-dhph"
server_key: "3d6a16bb32a59c4c080f7c51af674e46"
server_key_iv: "61333875ac55d1b9d293777b8bab78ff"
server_key_version: "encryptionkey@0"
I20260806 02:58:43.644660 1346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/minicluster-data/master-0/wal/instance:
uuid: "24ba9714cb1a494d983288e0ab1b4283"
format_stamp: "Formatted at 2026-08-06 02:58:43 on dist-test-slave-dhph"
server_key: "3d6a16bb32a59c4c080f7c51af674e46"
server_key_iv: "61333875ac55d1b9d293777b8bab78ff"
server_key_version: "encryptionkey@0"
I20260806 02:58:43.650876 1346 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260806 02:58:43.657138 1361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:58:43.659204 1346 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.001s
I20260806 02:58:43.659744 1346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/minicluster-data/master-0/wal
uuid: "24ba9714cb1a494d983288e0ab1b4283"
format_stamp: "Formatted at 2026-08-06 02:58:43 on dist-test-slave-dhph"
server_key: "3d6a16bb32a59c4c080f7c51af674e46"
server_key_iv: "61333875ac55d1b9d293777b8bab78ff"
server_key_version: "encryptionkey@0"
I20260806 02:58:43.660022 1346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:58:43.687047 1346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:58:43.692442 1346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:58:43.692984 1346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:58:43.724483 1346 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.254:42053
I20260806 02:58:43.724503 1422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.254:42053 every 8 connection(s)
I20260806 02:58:43.726131 1346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
I20260806 02:58:43.729990 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 1346
I20260806 02:58:43.730268 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/minicluster-data/master-0/wal/instance
I20260806 02:58:43.730934 1423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:58:43.731153 32043 external_mini_cluster.cc:1428] Setting key 17403c91188fb6662225567b854d646c
I20260806 02:58:43.748144 1423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24ba9714cb1a494d983288e0ab1b4283: Bootstrap starting.
Aug 06 02:58:43 dist-test-slave-dhph krb5kdc[1333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 02:58:43.752902 1423 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24ba9714cb1a494d983288e0ab1b4283: Neither blocks nor log segments found. Creating new log.
I20260806 02:58:43.754446 1423 log.cc:824] T 00000000000000000000000000000000 P 24ba9714cb1a494d983288e0ab1b4283: Log is configured to *not* fsync() on all Append() calls
I20260806 02:58:43.758251 1423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24ba9714cb1a494d983288e0ab1b4283: No bootstrap required, opened a new log
I20260806 02:58:43.762648 1426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:43.733920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36824 (local address 127.31.74.254:42053)
0806 02:58:43.735454 (+ 1534us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:43.735467 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:43.735524 (+ 57us) server_negotiation.cc:407] Connection header received
0806 02:58:43.738002 (+ 2478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:43.738032 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:43.738556 (+ 524us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:43.739093 (+ 537us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:43.741257 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:43.743012 (+ 1755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:43.745449 (+ 2437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:43.746484 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:43.746767 (+ 283us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:43.752583 (+ 5816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:58:43.752597 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:58:43.752622 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:58:43.752680 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:58:43.757063 (+ 4383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:43.758316 (+ 1253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:43.758327 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:43.758335 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:43.758563 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:43.759092 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:43.759096 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:43.759099 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:43.759820 (+ 721us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:58:43.760138 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:43.760623 (+ 485us) server_negotiation.cc:299] Negotiation successful
0806 02:58:43.761112 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":778,"thread_start_us":503,"threads_started":1}
I20260806 02:58:43.769568 1423 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24ba9714cb1a494d983288e0ab1b4283 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24ba9714cb1a494d983288e0ab1b4283" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 42053 } }
I20260806 02:58:43.769994 1423 raft_consensus.cc:374] T 00000000000000000000000000000000 P 24ba9714cb1a494d983288e0ab1b4283 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:58:43.770068 1423 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24ba9714cb1a494d983288e0ab1b4283 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24ba9714cb1a494d983288e0ab1b4283, State: Initialized, Role: FOLLOWER
I20260806 02:58:43.770840 1423 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24ba9714cb1a494d983288e0ab1b4283 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24ba9714cb1a494d983288e0ab1b4283" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 42053 } }
I20260806 02:58:43.771211 1423 raft_consensus.cc:388] T 00000000000000000000000000000000 P 24ba9714cb1a494d983288e0ab1b4283 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:58:43.771427 1423 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24ba9714cb1a494d983288e0ab1b4283 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:58:43.771744 1423 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24ba9714cb1a494d983288e0ab1b4283 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:58:43.774572 1423 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24ba9714cb1a494d983288e0ab1b4283 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24ba9714cb1a494d983288e0ab1b4283" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 42053 } }
I20260806 02:58:43.775274 1423 leader_election.cc:302] T 00000000000000000000000000000000 P 24ba9714cb1a494d983288e0ab1b4283 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 24ba9714cb1a494d983288e0ab1b4283; no voters:
I20260806 02:58:43.776068 1423 leader_election.cc:288] T 00000000000000000000000000000000 P 24ba9714cb1a494d983288e0ab1b4283 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:58:43.776283 1428 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24ba9714cb1a494d983288e0ab1b4283 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:58:43.777153 1428 raft_consensus.cc:686] T 00000000000000000000000000000000 P 24ba9714cb1a494d983288e0ab1b4283 [term 1 LEADER]: Becoming Leader. State: Replica: 24ba9714cb1a494d983288e0ab1b4283, State: Running, Role: LEADER
I20260806 02:58:43.778353 1428 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24ba9714cb1a494d983288e0ab1b4283 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24ba9714cb1a494d983288e0ab1b4283" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 42053 } }
I20260806 02:58:43.778721 1423 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24ba9714cb1a494d983288e0ab1b4283 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:58:43.782437 1429 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24ba9714cb1a494d983288e0ab1b4283 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "24ba9714cb1a494d983288e0ab1b4283" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24ba9714cb1a494d983288e0ab1b4283" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 42053 } } }
I20260806 02:58:43.783087 1429 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24ba9714cb1a494d983288e0ab1b4283 [sys.catalog]: This master's current role is: LEADER
I20260806 02:58:43.782485 1430 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24ba9714cb1a494d983288e0ab1b4283 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24ba9714cb1a494d983288e0ab1b4283. Latest consensus state: current_term: 1 leader_uuid: "24ba9714cb1a494d983288e0ab1b4283" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24ba9714cb1a494d983288e0ab1b4283" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 42053 } } }
I20260806 02:58:43.783908 1430 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24ba9714cb1a494d983288e0ab1b4283 [sys.catalog]: This master's current role is: LEADER
I20260806 02:58:43.787284 1438 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:58:43.793509 1438 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:58:43.803447 1438 catalog_manager.cc:1347] Generated new cluster ID: edc7f20a58f5442ab02188abe685a9d0
I20260806 02:58:43.803586 1438 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:58:43.810652 1438 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:58:43.812117 1438 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:58:43.821820 1438 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 24ba9714cb1a494d983288e0ab1b4283: Generated new TSK 0
I20260806 02:58:43.823300 1438 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.74.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.193.keytab.
Entry for principal kudu/127.31.74.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.193.keytab.
I20260806 02:58:43.914952 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.193:0
--local_ip_for_outbound_sockets=127.31.74.193
--webserver_interface=127.31.74.193
--webserver_port=0
--tserver_master_addrs=127.31.74.254:42053
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.193.keytab
--principal=kudu/127.31.74.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:46815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:58:44.347144 1450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:58:44.347463 1450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:58:44.347630 1450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:58:44.355410 1450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:58:44.355540 1450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:58:44.355686 1450 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260806 02:58:44.355751 1450 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 02:58:44.355861 1450 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.193
I20260806 02:58:44.367396 1450 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:46815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.193.keytab
--principal=kudu/127.31.74.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.74.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.74.254:42053
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.193
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:58:44.369169 1450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:58:44.370800 1450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:58:44.389155 1455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:58:44.389376 1456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:58:44.389648 1458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:58:44.390380 1450 server_base.cc:1034] running on GCE node
I20260806 02:58:44.391184 1450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:58:44.392617 1450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:58:44.393996 1450 hybrid_clock.cc:648] HybridClock initialized: now 1785985124393907 us; error 35 us; skew 500 ppm
I20260806 02:58:44.394707 1450 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:58:44 dist-test-slave-dhph krb5kdc[1333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985124, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:58:44.399163 1450 init.cc:377] Logged in from keytab as kudu/127.31.74.193@KRBTEST.COM (short username kudu)
I20260806 02:58:44.400960 1450 webserver.cc:460] Webserver started at http://127.31.74.193:41171/ using document root <none> and password file <none>
I20260806 02:58:44.402141 1450 fs_manager.cc:359] Metadata directory not provided
I20260806 02:58:44.402345 1450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:58:44.402922 1450 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:58:44.407157 1450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/minicluster-data/ts-0/data/instance:
uuid: "451b2658f7d946c8a4cc453e96d8613f"
format_stamp: "Formatted at 2026-08-06 02:58:44 on dist-test-slave-dhph"
server_key: "59340d1bae3b295ada75030e50c58c7a"
server_key_iv: "10954347204f3ad9625f055b0549549e"
server_key_version: "encryptionkey@0"
I20260806 02:58:44.408152 1450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/minicluster-data/ts-0/wal/instance:
uuid: "451b2658f7d946c8a4cc453e96d8613f"
format_stamp: "Formatted at 2026-08-06 02:58:44 on dist-test-slave-dhph"
server_key: "59340d1bae3b295ada75030e50c58c7a"
server_key_iv: "10954347204f3ad9625f055b0549549e"
server_key_version: "encryptionkey@0"
I20260806 02:58:44.415190 1450 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260806 02:58:44.419960 1465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:58:44.423259 1450 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.003s
I20260806 02:58:44.423519 1450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/minicluster-data/ts-0/wal
uuid: "451b2658f7d946c8a4cc453e96d8613f"
format_stamp: "Formatted at 2026-08-06 02:58:44 on dist-test-slave-dhph"
server_key: "59340d1bae3b295ada75030e50c58c7a"
server_key_iv: "10954347204f3ad9625f055b0549549e"
server_key_version: "encryptionkey@0"
I20260806 02:58:44.423761 1450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:58:44.438999 1450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:58:44.443857 1450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:58:44.444211 1450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:58:44.447873 1450 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:58:44.448022 1450 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:58:44.448180 1450 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:58:44.448276 1450 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 02:58:44 dist-test-slave-dhph krb5kdc[1333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985124, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.193@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 02:58:44.482743 1474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:44.449029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.193:54165 (local address 127.31.74.254:42053)
0806 02:58:44.449572 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:44.449579 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:44.450993 (+ 1414us) server_negotiation.cc:407] Connection header received
0806 02:58:44.453556 (+ 2563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:44.453565 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:44.453705 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:44.453870 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:44.456845 (+ 2975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:44.458347 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:44.461568 (+ 3221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:44.462333 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:44.462528 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:44.471934 (+ 9406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:58:44.471946 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:58:44.471956 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:58:44.472013 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:58:44.476413 (+ 4400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:44.477926 (+ 1513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:44.477931 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:44.477933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:44.478079 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:44.479688 (+ 1609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:44.479693 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:44.479695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:44.480212 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:58:44.480504 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:44.482065 (+ 1561us) server_negotiation.cc:299] Negotiation successful
0806 02:58:44.482339 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":264,"threads_started":1}
I20260806 02:58:44.485683 1475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:44.449795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:42053 (local address 127.31.74.193:54165)
0806 02:58:44.450773 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:44.450824 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 02:58:44.453093 (+ 2269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:58:44.454255 (+ 1162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:58:44.454265 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:58:44.455533 (+ 1268us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:58:44.456490 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:44.456522 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:44.458577 (+ 2055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:44.458586 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:44.461313 (+ 2727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:44.461358 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:44.463292 (+ 1934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:44.463396 (+ 104us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:44.463721 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:44.465386 (+ 1665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:58:44.465439 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:58:44.471483 (+ 6044us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:58:44.476673 (+ 5190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:58:44.476690 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:58:44.476720 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:58:44.477698 (+ 978us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:58:44.478452 (+ 754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:58:44.478460 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:58:44.478466 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:58:44.479479 (+ 1013us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:58:44.480454 (+ 975us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:58:44.480467 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:58:44.481214 (+ 747us) client_negotiation.cc:769] Sending connection context
0806 02:58:44.481894 (+ 680us) client_negotiation.cc:240] Negotiation successful
0806 02:58:44.482943 (+ 1049us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"thread_start_us":396,"threads_started":1}
I20260806 02:58:44.513398 1450 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.193:37965
I20260806 02:58:44.513451 1587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.193:37965 every 8 connection(s)
I20260806 02:58:44.515467 1450 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
I20260806 02:58:44.518219 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 1450
I20260806 02:58:44.518482 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/minicluster-data/ts-0/wal/instance
I20260806 02:58:44.518952 1588 heartbeater.cc:347] Connected to a master server at 127.31.74.254:42053
I20260806 02:58:44.519243 1588 heartbeater.cc:464] Registering TS with master...
I20260806 02:58:44.519452 32043 external_mini_cluster.cc:1428] Setting key 731e273184110370f05f29247aefa650
I20260806 02:58:44.519789 1474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:44.496458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.193:37381 (local address 127.31.74.254:42053)
0806 02:58:44.497108 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:44.497116 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:44.497815 (+ 699us) server_negotiation.cc:407] Connection header received
0806 02:58:44.498125 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:44.498133 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:44.498239 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:44.498381 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:44.499586 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:44.500808 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:44.504335 (+ 3527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:44.505857 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:44.506265 (+ 408us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:44.510175 (+ 3910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:58:44.510185 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:58:44.510190 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:58:44.510297 (+ 107us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:58:44.515516 (+ 5219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:44.516793 (+ 1277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:44.516804 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:44.516811 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:44.517059 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:44.517759 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:44.517763 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:44.517765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:44.518083 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:58:44.518221 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:44.519092 (+ 871us) server_negotiation.cc:299] Negotiation successful
0806 02:58:44.519400 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459}
I20260806 02:58:44.519847 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:44.496613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:42053 (local address 127.31.74.193:37381)
0806 02:58:44.497681 (+ 1068us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:44.497698 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:58:44.497941 (+ 243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:58:44.498379 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:58:44.498382 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:58:44.499079 (+ 697us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:58:44.499408 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:44.499418 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:44.501026 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:44.501032 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:44.504063 (+ 3031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:44.504096 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:44.506304 (+ 2208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:44.506320 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:44.506625 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:44.508085 (+ 1460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:58:44.508126 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:58:44.509793 (+ 1667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:58:44.515792 (+ 5999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:58:44.515808 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:58:44.515811 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:58:44.516547 (+ 736us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:58:44.517281 (+ 734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:58:44.517288 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:58:44.517292 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:58:44.517483 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:58:44.518241 (+ 758us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:58:44.518251 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:58:44.518522 (+ 271us) client_negotiation.cc:769] Sending connection context
0806 02:58:44.518742 (+ 220us) client_negotiation.cc:240] Negotiation successful
0806 02:58:44.519200 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":912,"thread_start_us":780,"threads_started":1}
I20260806 02:58:44.520524 1588 heartbeater.cc:511] Master 127.31.74.254:42053 requested a full tablet report, sending...
I20260806 02:58:44.523250 1378 ts_manager.cc:194] Registered new tserver with Master: 451b2658f7d946c8a4cc453e96d8613f (127.31.74.193:37965)
I20260806 02:58:44.526136 1378 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.74.193@KRBTEST.COM'} at 127.31.74.193:54165
WARNING: no policy specified for kudu/127.31.74.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.194.keytab.
Entry for principal kudu/127.31.74.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.194.keytab.
I20260806 02:58:44.594947 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.194:0
--local_ip_for_outbound_sockets=127.31.74.194
--webserver_interface=127.31.74.194
--webserver_port=0
--tserver_master_addrs=127.31.74.254:42053
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.194.keytab
--principal=kudu/127.31.74.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:46815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:58:45.025106 1592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:58:45.025817 1592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:58:45.026244 1592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:58:45.038453 1592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:58:45.038653 1592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:58:45.038763 1592 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260806 02:58:45.038826 1592 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 02:58:45.038897 1592 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.194
I20260806 02:58:45.054316 1592 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:46815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.194.keytab
--principal=kudu/127.31.74.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.74.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.74.254:42053
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.194
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:58:45.055784 1592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:58:45.057148 1592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:58:45.074317 1597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:58:45.074674 1598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:58:45.074698 1600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:58:45.076172 1592 server_base.cc:1034] running on GCE node
I20260806 02:58:45.078105 1592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:58:45.079612 1592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:58:45.080906 1592 hybrid_clock.cc:648] HybridClock initialized: now 1785985125080826 us; error 83 us; skew 500 ppm
I20260806 02:58:45.081669 1592 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:58:45 dist-test-slave-dhph krb5kdc[1333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985125, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:58:45.087378 1592 init.cc:377] Logged in from keytab as kudu/127.31.74.194@KRBTEST.COM (short username kudu)
I20260806 02:58:45.089787 1592 webserver.cc:460] Webserver started at http://127.31.74.194:41571/ using document root <none> and password file <none>
I20260806 02:58:45.090880 1592 fs_manager.cc:359] Metadata directory not provided
I20260806 02:58:45.091050 1592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:58:45.091428 1592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:58:45.094939 1592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/minicluster-data/ts-1/data/instance:
uuid: "c2d58cba7d4847c4901dcd119b6bd6bb"
format_stamp: "Formatted at 2026-08-06 02:58:45 on dist-test-slave-dhph"
server_key: "604dac0e54a4e40365af7147b5b40d5a"
server_key_iv: "cc50daed60688444ab6f3a6161bf6579"
server_key_version: "encryptionkey@0"
I20260806 02:58:45.095795 1592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/minicluster-data/ts-1/wal/instance:
uuid: "c2d58cba7d4847c4901dcd119b6bd6bb"
format_stamp: "Formatted at 2026-08-06 02:58:45 on dist-test-slave-dhph"
server_key: "604dac0e54a4e40365af7147b5b40d5a"
server_key_iv: "cc50daed60688444ab6f3a6161bf6579"
server_key_version: "encryptionkey@0"
I20260806 02:58:45.102027 1592 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260806 02:58:45.108560 1607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:58:45.110983 1592 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.006s sys 0.000s
I20260806 02:58:45.111451 1592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/minicluster-data/ts-1/wal
uuid: "c2d58cba7d4847c4901dcd119b6bd6bb"
format_stamp: "Formatted at 2026-08-06 02:58:45 on dist-test-slave-dhph"
server_key: "604dac0e54a4e40365af7147b5b40d5a"
server_key_iv: "cc50daed60688444ab6f3a6161bf6579"
server_key_version: "encryptionkey@0"
I20260806 02:58:45.111944 1592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:58:45.136487 1592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:58:45.143213 1592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:58:45.143869 1592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:58:45.148193 1592 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:58:45.148644 1592 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:58:45.148840 1592 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:58:45.148932 1592 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 02:58:45 dist-test-slave-dhph krb5kdc[1333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985125, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.194@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 02:58:45.191601 1615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:45.148837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.194:41737 (local address 127.31.74.254:42053)
0806 02:58:45.149442 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:45.149451 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:45.151727 (+ 2276us) server_negotiation.cc:407] Connection header received
0806 02:58:45.154458 (+ 2731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:45.154465 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:45.154639 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:45.154780 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:45.158001 (+ 3221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:45.159379 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:45.166741 (+ 7362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:45.167493 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:45.167827 (+ 334us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:45.175982 (+ 8155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:58:45.175993 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:58:45.175999 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:58:45.176057 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:58:45.183632 (+ 7575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:45.185539 (+ 1907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:45.185545 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:45.185547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:45.185769 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:45.187003 (+ 1234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:45.187010 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:45.187014 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:45.187446 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:58:45.187669 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:45.190915 (+ 3246us) server_negotiation.cc:299] Negotiation successful
0806 02:58:45.191176 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":293,"threads_started":1}
I20260806 02:58:45.193879 1617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:45.149774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:42053 (local address 127.31.74.194:41737)
0806 02:58:45.151165 (+ 1391us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:45.151456 (+ 291us) client_negotiation.cc:174] Beginning negotiation
0806 02:58:45.154013 (+ 2557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:58:45.155093 (+ 1080us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:58:45.155103 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:58:45.156593 (+ 1490us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:58:45.157706 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:45.157740 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:45.159657 (+ 1917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:45.159664 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:45.165536 (+ 5872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:45.165576 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:45.168162 (+ 2586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:45.168179 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:45.169001 (+ 822us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:45.170757 (+ 1756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:58:45.170850 (+ 93us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:58:45.175526 (+ 4676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:58:45.183872 (+ 8346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:58:45.183886 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:58:45.183908 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:58:45.185320 (+ 1412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:58:45.185966 (+ 646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:58:45.185974 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:58:45.185980 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:58:45.186756 (+ 776us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:58:45.187904 (+ 1148us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:58:45.187925 (+ 21us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:58:45.189561 (+ 1636us) client_negotiation.cc:769] Sending connection context
0806 02:58:45.190314 (+ 753us) client_negotiation.cc:240] Negotiation successful
0806 02:58:45.191428 (+ 1114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":905,"thread_start_us":508,"threads_started":1}
I20260806 02:58:45.233166 1592 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.194:36655
I20260806 02:58:45.233830 1730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.194:36655 every 8 connection(s)
I20260806 02:58:45.234869 1592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
I20260806 02:58:45.237566 1698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:45.208493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:42053 (local address 127.31.74.194:33533)
0806 02:58:45.211486 (+ 2993us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:45.211518 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0806 02:58:45.211737 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:58:45.212398 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:58:45.212407 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:58:45.214688 (+ 2281us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:58:45.217462 (+ 2774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:45.217497 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:45.219578 (+ 2081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:45.219590 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:45.221231 (+ 1641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:45.221257 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:45.223420 (+ 2163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:45.223612 (+ 192us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:45.223953 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:45.225004 (+ 1051us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:58:45.225038 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:58:45.227237 (+ 2199us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:58:45.232770 (+ 5533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:58:45.232779 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:58:45.232784 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:58:45.233635 (+ 851us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:58:45.234556 (+ 921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:58:45.234562 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:58:45.234567 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:58:45.234789 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:58:45.235744 (+ 955us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:58:45.235749 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:58:45.236269 (+ 520us) client_negotiation.cc:769] Sending connection context
0806 02:58:45.236459 (+ 190us) client_negotiation.cc:240] Negotiation successful
0806 02:58:45.237040 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2756,"thread_start_us":2572,"threads_started":1}
I20260806 02:58:45.238189 1731 heartbeater.cc:347] Connected to a master server at 127.31.74.254:42053
I20260806 02:58:45.238497 1731 heartbeater.cc:464] Registering TS with master...
I20260806 02:58:45.239269 1615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:45.208560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.194:33533 (local address 127.31.74.254:42053)
0806 02:58:45.208847 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:45.208858 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:45.211673 (+ 2815us) server_negotiation.cc:407] Connection header received
0806 02:58:45.211905 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:45.211913 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:45.212030 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:45.212161 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:45.217784 (+ 5623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:45.219331 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:45.221490 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:45.223026 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:45.223324 (+ 298us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:45.227637 (+ 4313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:58:45.227661 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:58:45.227668 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:58:45.227728 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:58:45.232474 (+ 4746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:45.233834 (+ 1360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:45.233841 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:45.233848 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:45.234047 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:45.234996 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:45.235002 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:45.235005 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:45.235541 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:58:45.237812 (+ 2271us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:45.238429 (+ 617us) server_negotiation.cc:299] Negotiation successful
0806 02:58:45.238828 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260806 02:58:45.239431 1731 heartbeater.cc:511] Master 127.31.74.254:42053 requested a full tablet report, sending...
I20260806 02:58:45.241595 1378 ts_manager.cc:194] Registered new tserver with Master: c2d58cba7d4847c4901dcd119b6bd6bb (127.31.74.194:36655)
I20260806 02:58:45.242553 1378 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.74.194@KRBTEST.COM'} at 127.31.74.194:41737
I20260806 02:58:45.243306 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 1592
I20260806 02:58:45.243511 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/minicluster-data/ts-1/wal/instance
I20260806 02:58:45.244094 32043 external_mini_cluster.cc:1428] Setting key 4a6786247e8ece294f855b6d9f9e2770
WARNING: no policy specified for kudu/127.31.74.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.195.keytab.
Entry for principal kudu/127.31.74.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.195.keytab.
I20260806 02:58:45.315490 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.195:0
--local_ip_for_outbound_sockets=127.31.74.195
--webserver_interface=127.31.74.195
--webserver_port=0
--tserver_master_addrs=127.31.74.254:42053
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.195.keytab
--principal=kudu/127.31.74.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:46815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:58:45.531673 1588 heartbeater.cc:503] Master 127.31.74.254:42053 was elected leader, sending a full tablet report...
W20260806 02:58:45.778733 1735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:58:45.779142 1735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:58:45.779269 1735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:58:45.789969 1735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:58:45.790122 1735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:58:45.790222 1735 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260806 02:58:45.790508 1735 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 02:58:45.790724 1735 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.195
I20260806 02:58:45.805016 1735 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:46815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.195.keytab
--principal=kudu/127.31.74.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.74.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.74.254:42053
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.195
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:58:45.806684 1735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:58:45.808215 1735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:58:45.826793 1743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:58:45.826967 1740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:58:45.826967 1741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:58:45.827564 1735 server_base.cc:1034] running on GCE node
I20260806 02:58:45.829107 1735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:58:45.830956 1735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:58:45.832280 1735 hybrid_clock.cc:648] HybridClock initialized: now 1785985125832230 us; error 64 us; skew 500 ppm
I20260806 02:58:45.832687 1735 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:58:45 dist-test-slave-dhph krb5kdc[1333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985125, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:58:45.836971 1735 init.cc:377] Logged in from keytab as kudu/127.31.74.195@KRBTEST.COM (short username kudu)
I20260806 02:58:45.839164 1735 webserver.cc:460] Webserver started at http://127.31.74.195:36333/ using document root <none> and password file <none>
I20260806 02:58:45.840003 1735 fs_manager.cc:359] Metadata directory not provided
I20260806 02:58:45.840185 1735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:58:45.840552 1735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:58:45.845638 1735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/minicluster-data/ts-2/data/instance:
uuid: "f0ac4bacaea845ad987b992b68df2772"
format_stamp: "Formatted at 2026-08-06 02:58:45 on dist-test-slave-dhph"
server_key: "a01a4e1868df022c1c6656606f033a7b"
server_key_iv: "d6c4ad4b199a69dce44f75d798c83d9b"
server_key_version: "encryptionkey@0"
I20260806 02:58:45.847185 1735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/minicluster-data/ts-2/wal/instance:
uuid: "f0ac4bacaea845ad987b992b68df2772"
format_stamp: "Formatted at 2026-08-06 02:58:45 on dist-test-slave-dhph"
server_key: "a01a4e1868df022c1c6656606f033a7b"
server_key_iv: "d6c4ad4b199a69dce44f75d798c83d9b"
server_key_version: "encryptionkey@0"
I20260806 02:58:45.856185 1735 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.001s
I20260806 02:58:45.863075 1750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:58:45.867019 1735 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.006s sys 0.000s
I20260806 02:58:45.867249 1735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/minicluster-data/ts-2/wal
uuid: "f0ac4bacaea845ad987b992b68df2772"
format_stamp: "Formatted at 2026-08-06 02:58:45 on dist-test-slave-dhph"
server_key: "a01a4e1868df022c1c6656606f033a7b"
server_key_iv: "d6c4ad4b199a69dce44f75d798c83d9b"
server_key_version: "encryptionkey@0"
I20260806 02:58:45.867537 1735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:58:45.889282 1735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:58:45.898864 1735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:58:45.899258 1735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:58:45.904376 1735 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:58:45.904588 1735 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:58:45.904835 1735 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:58:45.905030 1735 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 02:58:45 dist-test-slave-dhph krb5kdc[1333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985125, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.195@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 02:58:45.941692 1759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:45.904750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.195:34047 (local address 127.31.74.254:42053)
0806 02:58:45.905691 (+ 941us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:45.905698 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:45.907758 (+ 2060us) server_negotiation.cc:407] Connection header received
0806 02:58:45.910703 (+ 2945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:45.910712 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:45.910845 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:45.910941 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:45.916167 (+ 5226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:45.917948 (+ 1781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:45.922648 (+ 4700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:45.923369 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:45.923544 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:45.929266 (+ 5722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:58:45.929272 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:58:45.929329 (+ 57us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:58:45.929373 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:58:45.935365 (+ 5992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:45.937003 (+ 1638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:45.937014 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:45.937017 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:45.937187 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:45.938179 (+ 992us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:45.938185 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:45.938187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:45.938933 (+ 746us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:58:45.939155 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:45.941088 (+ 1933us) server_negotiation.cc:299] Negotiation successful
0806 02:58:45.941344 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":691,"thread_start_us":479,"threads_started":1}
I20260806 02:58:45.946520 1761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:45.906243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:42053 (local address 127.31.74.195:34047)
0806 02:58:45.907524 (+ 1281us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:45.907585 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0806 02:58:45.910150 (+ 2565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:58:45.911938 (+ 1788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:58:45.911958 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:58:45.913477 (+ 1519us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:58:45.915726 (+ 2249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:45.915858 (+ 132us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:45.918201 (+ 2343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:45.918206 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:45.922209 (+ 4003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:45.922264 (+ 55us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:45.923683 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:45.923692 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:45.923986 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:45.925003 (+ 1017us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:58:45.925048 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:58:45.928948 (+ 3900us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:58:45.935781 (+ 6833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:58:45.935803 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:58:45.935825 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:58:45.936783 (+ 958us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:58:45.937349 (+ 566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:58:45.937353 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:58:45.937356 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:58:45.937844 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:58:45.939187 (+ 1343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:58:45.939200 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:58:45.939943 (+ 743us) client_negotiation.cc:769] Sending connection context
0806 02:58:45.940921 (+ 978us) client_negotiation.cc:240] Negotiation successful
0806 02:58:45.942494 (+ 1573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":944,"thread_start_us":566,"threads_started":1}
I20260806 02:58:45.984423 1735 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.195:39003
I20260806 02:58:45.984449 1872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.195:39003 every 8 connection(s)
I20260806 02:58:45.989472 1735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
I20260806 02:58:45.989988 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 1735
I20260806 02:58:45.990257 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/minicluster-data/ts-2/wal/instance
I20260806 02:58:45.990480 1759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:45.958941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.195:39689 (local address 127.31.74.254:42053)
0806 02:58:45.959215 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:45.959230 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:45.967780 (+ 8550us) server_negotiation.cc:407] Connection header received
0806 02:58:45.968033 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:45.968039 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:45.968130 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:45.968244 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:45.970182 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:45.971487 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:45.974082 (+ 2595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:45.974719 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:45.974892 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:45.977510 (+ 2618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:58:45.977516 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:58:45.977519 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:58:45.977560 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:58:45.982966 (+ 5406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:45.984840 (+ 1874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:45.984847 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:45.984851 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:45.985055 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:45.985602 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:45.985608 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:45.985611 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:45.986626 (+ 1015us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:58:45.987054 (+ 428us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:45.989169 (+ 2115us) server_negotiation.cc:299] Negotiation successful
0806 02:58:45.989510 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260806 02:58:45.990517 1847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:45.959136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:42053 (local address 127.31.74.195:39689)
0806 02:58:45.967471 (+ 8335us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:45.967514 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 02:58:45.967777 (+ 263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:58:45.968326 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:58:45.968333 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:58:45.969276 (+ 943us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:58:45.969872 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:45.969896 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:45.971724 (+ 1828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:45.971729 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:45.973843 (+ 2114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:45.973867 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:45.975152 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:45.975162 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:45.975319 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:45.976075 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:58:45.976096 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:58:45.977215 (+ 1119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:58:45.983233 (+ 6018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:58:45.983243 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:58:45.983250 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:58:45.984595 (+ 1345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:58:45.985279 (+ 684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:58:45.985284 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:58:45.985286 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:58:45.985457 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:58:45.987095 (+ 1638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:58:45.987100 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:58:45.988066 (+ 966us) client_negotiation.cc:769] Sending connection context
0806 02:58:45.988658 (+ 592us) client_negotiation.cc:240] Negotiation successful
0806 02:58:45.989791 (+ 1133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8060,"thread_start_us":7812,"threads_started":1}
I20260806 02:58:45.990981 32043 external_mini_cluster.cc:1428] Setting key 8a30643242f52806364c7c4a45291051
I20260806 02:58:45.993492 1873 heartbeater.cc:347] Connected to a master server at 127.31.74.254:42053
I20260806 02:58:45.993839 1873 heartbeater.cc:464] Registering TS with master...
I20260806 02:58:45.994879 1873 heartbeater.cc:511] Master 127.31.74.254:42053 requested a full tablet report, sending...
I20260806 02:58:45.998071 1376 ts_manager.cc:194] Registered new tserver with Master: f0ac4bacaea845ad987b992b68df2772 (127.31.74.195:39003)
I20260806 02:58:45.999356 1376 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.74.195@KRBTEST.COM'} at 127.31.74.195:34047
I20260806 02:58:46.012584 32043 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 02:58:46 dist-test-slave-dhph krb5kdc[1333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985126, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 02:58:46 dist-test-slave-dhph krb5kdc[1333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985126, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 02:58:46.091135 1759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:46.071681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36838 (local address 127.31.74.254:42053)
0806 02:58:46.072093 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:46.072101 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:46.072370 (+ 269us) server_negotiation.cc:407] Connection header received
0806 02:58:46.072607 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:46.072613 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:46.072695 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:46.072798 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:46.074656 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:46.075571 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:46.077290 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:46.077851 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:46.078261 (+ 410us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:46.083482 (+ 5221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:58:46.083489 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:58:46.083492 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:58:46.083540 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:58:46.087728 (+ 4188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:46.088664 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:46.088670 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:46.088674 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:46.088910 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:46.089339 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:46.089347 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:46.089351 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:46.089693 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:58:46.090037 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:46.090392 (+ 355us) server_negotiation.cc:299] Negotiation successful
0806 02:58:46.090689 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260806 02:58:46.096414 1376 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36838:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:58:46.105257 1376 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:58:46.156767 1889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:46.121216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:37965 (local address 127.0.0.1:36948)
0806 02:58:46.142516 (+ 21300us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:46.142623 (+ 107us) client_negotiation.cc:174] Beginning negotiation
0806 02:58:46.145232 (+ 2609us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:58:46.147381 (+ 2149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:58:46.147385 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:58:46.147626 (+ 241us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:58:46.148480 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:46.148500 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:46.150828 (+ 2328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:46.150835 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:46.153346 (+ 2511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:46.153361 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:46.155486 (+ 2125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:46.155494 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:46.155734 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:46.155793 (+ 59us) client_negotiation.cc:769] Sending connection context
0806 02:58:46.155957 (+ 164us) client_negotiation.cc:240] Negotiation successful
0806 02:58:46.156221 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20814,"mutex_wait_us":79,"spinlock_wait_cycles":470144,"thread_start_us":23638,"threads_started":1}
I20260806 02:58:46.157075 1890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:46.122637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37248 (local address 127.31.74.194:36655)
0806 02:58:46.130754 (+ 8117us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:46.130791 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:46.145396 (+ 14605us) server_negotiation.cc:407] Connection header received
0806 02:58:46.146003 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:46.146111 (+ 108us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:46.146665 (+ 554us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:46.146896 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:58:46.148967 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:46.151037 (+ 2070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:46.154162 (+ 3125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:46.155811 (+ 1649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:46.155987 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:46.156208 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:46.156551 (+ 343us) server_negotiation.cc:299] Negotiation successful
0806 02:58:46.156751 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7436,"thread_start_us":7258,"threads_started":1}
I20260806 02:58:46.157744 1887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:46.121487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36948 (local address 127.31.74.193:37965)
0806 02:58:46.146667 (+ 25180us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:46.146677 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:46.146758 (+ 81us) server_negotiation.cc:407] Connection header received
0806 02:58:46.146859 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:46.146866 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:46.147161 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:46.147353 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:58:46.148716 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:46.150522 (+ 1806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:46.153546 (+ 3024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:46.155196 (+ 1650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:46.155377 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:46.155626 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:46.157067 (+ 1441us) server_negotiation.cc:299] Negotiation successful
0806 02:58:46.157374 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24847,"thread_start_us":24347,"threads_started":1}
I20260806 02:58:46.158540 1888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:46.122244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.195:39003 (local address 127.0.0.1:60902)
0806 02:58:46.148022 (+ 25778us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:46.148047 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 02:58:46.148208 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:58:46.149211 (+ 1003us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:58:46.149216 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:58:46.149246 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:58:46.149596 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:46.149607 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:46.152050 (+ 2443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:46.152057 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:46.154422 (+ 2365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:46.154433 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:46.157352 (+ 2919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:46.157360 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:46.157563 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:46.157604 (+ 41us) client_negotiation.cc:769] Sending connection context
0806 02:58:46.157713 (+ 109us) client_negotiation.cc:240] Negotiation successful
0806 02:58:46.158091 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25590,"thread_start_us":18965,"threads_started":1}
I20260806 02:58:46.158630 1886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:46.121793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.194:36655 (local address 127.0.0.1:37248)
0806 02:58:46.145209 (+ 23416us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:46.145233 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 02:58:46.145550 (+ 317us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:58:46.147216 (+ 1666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:58:46.147224 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:58:46.147668 (+ 444us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:58:46.148536 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:46.148547 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:46.151300 (+ 2753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:46.151306 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:46.153949 (+ 2643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:46.153970 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:46.156061 (+ 2091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:46.156068 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:46.156284 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:46.156335 (+ 51us) client_negotiation.cc:769] Sending connection context
0806 02:58:46.157413 (+ 1078us) client_negotiation.cc:240] Negotiation successful
0806 02:58:46.157571 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23261,"mutex_wait_us":50,"thread_start_us":23814,"threads_started":1}
I20260806 02:58:46.159087 1891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:46.123008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60902 (local address 127.31.74.195:39003)
0806 02:58:46.144160 (+ 21152us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:46.144176 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:46.148200 (+ 4024us) server_negotiation.cc:407] Connection header received
0806 02:58:46.148398 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:46.148407 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:46.148697 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:46.149219 (+ 522us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:58:46.149896 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:46.151867 (+ 1971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:46.154643 (+ 2776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:46.156511 (+ 1868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:46.156660 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:46.158225 (+ 1565us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:46.158321 (+ 96us) server_negotiation.cc:299] Negotiation successful
0806 02:58:46.158513 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20863,"thread_start_us":21327,"threads_started":1}
I20260806 02:58:46.162907 1802 tablet_service.cc:1467] Processing CreateTablet for tablet 34dee7f4687243aaa359bda73a875b04 (DEFAULT_TABLE table=test-table [id=48cb84502c134ca2998fd8edb9098420]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:58:46.162874 1517 tablet_service.cc:1467] Processing CreateTablet for tablet 34dee7f4687243aaa359bda73a875b04 (DEFAULT_TABLE table=test-table [id=48cb84502c134ca2998fd8edb9098420]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:58:46.164475 1802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34dee7f4687243aaa359bda73a875b04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:58:46.164552 1517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34dee7f4687243aaa359bda73a875b04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:58:46.165846 1659 tablet_service.cc:1467] Processing CreateTablet for tablet 34dee7f4687243aaa359bda73a875b04 (DEFAULT_TABLE table=test-table [id=48cb84502c134ca2998fd8edb9098420]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:58:46.167971 1659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34dee7f4687243aaa359bda73a875b04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:58:46.181380 1892 tablet_bootstrap.cc:492] T 34dee7f4687243aaa359bda73a875b04 P f0ac4bacaea845ad987b992b68df2772: Bootstrap starting.
I20260806 02:58:46.183166 1893 tablet_bootstrap.cc:492] T 34dee7f4687243aaa359bda73a875b04 P 451b2658f7d946c8a4cc453e96d8613f: Bootstrap starting.
I20260806 02:58:46.186105 1892 tablet_bootstrap.cc:654] T 34dee7f4687243aaa359bda73a875b04 P f0ac4bacaea845ad987b992b68df2772: Neither blocks nor log segments found. Creating new log.
I20260806 02:58:46.188557 1893 tablet_bootstrap.cc:654] T 34dee7f4687243aaa359bda73a875b04 P 451b2658f7d946c8a4cc453e96d8613f: Neither blocks nor log segments found. Creating new log.
I20260806 02:58:46.189364 1892 log.cc:824] T 34dee7f4687243aaa359bda73a875b04 P f0ac4bacaea845ad987b992b68df2772: Log is configured to *not* fsync() on all Append() calls
I20260806 02:58:46.189997 1894 tablet_bootstrap.cc:492] T 34dee7f4687243aaa359bda73a875b04 P c2d58cba7d4847c4901dcd119b6bd6bb: Bootstrap starting.
I20260806 02:58:46.190430 1893 log.cc:824] T 34dee7f4687243aaa359bda73a875b04 P 451b2658f7d946c8a4cc453e96d8613f: Log is configured to *not* fsync() on all Append() calls
I20260806 02:58:46.195169 1894 tablet_bootstrap.cc:654] T 34dee7f4687243aaa359bda73a875b04 P c2d58cba7d4847c4901dcd119b6bd6bb: Neither blocks nor log segments found. Creating new log.
I20260806 02:58:46.195247 1893 tablet_bootstrap.cc:492] T 34dee7f4687243aaa359bda73a875b04 P 451b2658f7d946c8a4cc453e96d8613f: No bootstrap required, opened a new log
I20260806 02:58:46.195487 1892 tablet_bootstrap.cc:492] T 34dee7f4687243aaa359bda73a875b04 P f0ac4bacaea845ad987b992b68df2772: No bootstrap required, opened a new log
I20260806 02:58:46.195770 1893 ts_tablet_manager.cc:1397] T 34dee7f4687243aaa359bda73a875b04 P 451b2658f7d946c8a4cc453e96d8613f: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.009s
I20260806 02:58:46.195820 1892 ts_tablet_manager.cc:1397] T 34dee7f4687243aaa359bda73a875b04 P f0ac4bacaea845ad987b992b68df2772: Time spent bootstrapping tablet: real 0.015s user 0.011s sys 0.000s
I20260806 02:58:46.197160 1894 log.cc:824] T 34dee7f4687243aaa359bda73a875b04 P c2d58cba7d4847c4901dcd119b6bd6bb: Log is configured to *not* fsync() on all Append() calls
I20260806 02:58:46.202613 1894 tablet_bootstrap.cc:492] T 34dee7f4687243aaa359bda73a875b04 P c2d58cba7d4847c4901dcd119b6bd6bb: No bootstrap required, opened a new log
I20260806 02:58:46.203042 1894 ts_tablet_manager.cc:1397] T 34dee7f4687243aaa359bda73a875b04 P c2d58cba7d4847c4901dcd119b6bd6bb: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.004s
I20260806 02:58:46.205756 1893 raft_consensus.cc:348] T 34dee7f4687243aaa359bda73a875b04 P 451b2658f7d946c8a4cc453e96d8613f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451b2658f7d946c8a4cc453e96d8613f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37965 } } peers { permanent_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 36655 } } peers { permanent_uuid: "f0ac4bacaea845ad987b992b68df2772" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 39003 } }
I20260806 02:58:46.206691 1893 raft_consensus.cc:374] T 34dee7f4687243aaa359bda73a875b04 P 451b2658f7d946c8a4cc453e96d8613f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:58:46.206883 1893 raft_consensus.cc:729] T 34dee7f4687243aaa359bda73a875b04 P 451b2658f7d946c8a4cc453e96d8613f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 451b2658f7d946c8a4cc453e96d8613f, State: Initialized, Role: FOLLOWER
I20260806 02:58:46.207701 1893 consensus_queue.cc:260] T 34dee7f4687243aaa359bda73a875b04 P 451b2658f7d946c8a4cc453e96d8613f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451b2658f7d946c8a4cc453e96d8613f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37965 } } peers { permanent_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 36655 } } peers { permanent_uuid: "f0ac4bacaea845ad987b992b68df2772" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 39003 } }
I20260806 02:58:46.210419 1893 ts_tablet_manager.cc:1428] T 34dee7f4687243aaa359bda73a875b04 P 451b2658f7d946c8a4cc453e96d8613f: Time spent starting tablet: real 0.014s user 0.009s sys 0.005s
I20260806 02:58:46.211498 1892 raft_consensus.cc:348] T 34dee7f4687243aaa359bda73a875b04 P f0ac4bacaea845ad987b992b68df2772 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451b2658f7d946c8a4cc453e96d8613f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37965 } } peers { permanent_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 36655 } } peers { permanent_uuid: "f0ac4bacaea845ad987b992b68df2772" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 39003 } }
I20260806 02:58:46.212253 1892 raft_consensus.cc:374] T 34dee7f4687243aaa359bda73a875b04 P f0ac4bacaea845ad987b992b68df2772 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:58:46.212383 1892 raft_consensus.cc:729] T 34dee7f4687243aaa359bda73a875b04 P f0ac4bacaea845ad987b992b68df2772 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0ac4bacaea845ad987b992b68df2772, State: Initialized, Role: FOLLOWER
I20260806 02:58:46.213208 1892 consensus_queue.cc:260] T 34dee7f4687243aaa359bda73a875b04 P f0ac4bacaea845ad987b992b68df2772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451b2658f7d946c8a4cc453e96d8613f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37965 } } peers { permanent_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 36655 } } peers { permanent_uuid: "f0ac4bacaea845ad987b992b68df2772" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 39003 } }
I20260806 02:58:46.214956 1873 heartbeater.cc:503] Master 127.31.74.254:42053 was elected leader, sending a full tablet report...
I20260806 02:58:46.215629 1892 ts_tablet_manager.cc:1428] T 34dee7f4687243aaa359bda73a875b04 P f0ac4bacaea845ad987b992b68df2772: Time spent starting tablet: real 0.020s user 0.008s sys 0.009s
I20260806 02:58:46.217288 1894 raft_consensus.cc:348] T 34dee7f4687243aaa359bda73a875b04 P c2d58cba7d4847c4901dcd119b6bd6bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451b2658f7d946c8a4cc453e96d8613f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37965 } } peers { permanent_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 36655 } } peers { permanent_uuid: "f0ac4bacaea845ad987b992b68df2772" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 39003 } }
I20260806 02:58:46.217900 1894 raft_consensus.cc:374] T 34dee7f4687243aaa359bda73a875b04 P c2d58cba7d4847c4901dcd119b6bd6bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:58:46.218462 1894 raft_consensus.cc:729] T 34dee7f4687243aaa359bda73a875b04 P c2d58cba7d4847c4901dcd119b6bd6bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2d58cba7d4847c4901dcd119b6bd6bb, State: Initialized, Role: FOLLOWER
I20260806 02:58:46.219290 1894 consensus_queue.cc:260] T 34dee7f4687243aaa359bda73a875b04 P c2d58cba7d4847c4901dcd119b6bd6bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451b2658f7d946c8a4cc453e96d8613f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37965 } } peers { permanent_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 36655 } } peers { permanent_uuid: "f0ac4bacaea845ad987b992b68df2772" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 39003 } }
I20260806 02:58:46.220665 1731 heartbeater.cc:503] Master 127.31.74.254:42053 was elected leader, sending a full tablet report...
I20260806 02:58:46.221549 1894 ts_tablet_manager.cc:1428] T 34dee7f4687243aaa359bda73a875b04 P c2d58cba7d4847c4901dcd119b6bd6bb: Time spent starting tablet: real 0.018s user 0.017s sys 0.000s
W20260806 02:58:46.239058 1732 tablet.cc:2369] T 34dee7f4687243aaa359bda73a875b04 P c2d58cba7d4847c4901dcd119b6bd6bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:58:46.242319 1874 tablet.cc:2369] T 34dee7f4687243aaa359bda73a875b04 P f0ac4bacaea845ad987b992b68df2772: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:58:46.269729 1589 tablet.cc:2369] T 34dee7f4687243aaa359bda73a875b04 P 451b2658f7d946c8a4cc453e96d8613f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:58:46.418072 1898 raft_consensus.cc:482] T 34dee7f4687243aaa359bda73a875b04 P 451b2658f7d946c8a4cc453e96d8613f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:58:46.418416 1898 raft_consensus.cc:504] T 34dee7f4687243aaa359bda73a875b04 P 451b2658f7d946c8a4cc453e96d8613f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451b2658f7d946c8a4cc453e96d8613f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37965 } } peers { permanent_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 36655 } } peers { permanent_uuid: "f0ac4bacaea845ad987b992b68df2772" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 39003 } }
I20260806 02:58:46.420780 1898 leader_election.cc:288] T 34dee7f4687243aaa359bda73a875b04 P 451b2658f7d946c8a4cc453e96d8613f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2d58cba7d4847c4901dcd119b6bd6bb (127.31.74.194:36655), f0ac4bacaea845ad987b992b68df2772 (127.31.74.195:39003)
I20260806 02:58:46.433327 1891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:46.421389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.193:54267 (local address 127.31.74.195:39003)
0806 02:58:46.421853 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:46.421861 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:46.422686 (+ 825us) server_negotiation.cc:407] Connection header received
0806 02:58:46.422979 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:46.422986 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:46.423077 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:46.423197 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:58:46.424255 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:46.425533 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:46.429882 (+ 4349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:46.431409 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:46.431709 (+ 300us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:46.431800 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:46.432893 (+ 1093us) server_negotiation.cc:299] Negotiation successful
0806 02:58:46.433080 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 02:58:46.433681 1902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:46.421171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.194:36655 (local address 127.31.74.193:45975)
0806 02:58:46.422522 (+ 1351us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:46.422550 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0806 02:58:46.422752 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:58:46.423357 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:58:46.423363 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:58:46.423416 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:58:46.424058 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:46.424076 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:46.425838 (+ 1762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:46.425847 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:46.429498 (+ 3651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:46.429528 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:46.431998 (+ 2470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:46.432008 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:46.432442 (+ 434us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:46.432628 (+ 186us) client_negotiation.cc:769] Sending connection context
0806 02:58:46.432788 (+ 160us) client_negotiation.cc:240] Negotiation successful
0806 02:58:46.433059 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1175,"mutex_wait_us":548,"spinlock_wait_cycles":16000,"thread_start_us":911,"threads_started":1}
I20260806 02:58:46.433705 1901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:46.421311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.195:39003 (local address 127.31.74.193:54267)
0806 02:58:46.422522 (+ 1211us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:46.422544 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 02:58:46.422791 (+ 247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:58:46.423357 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:58:46.423363 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:58:46.423418 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:58:46.424076 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:46.424085 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:46.425838 (+ 1753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:46.425848 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:46.429498 (+ 3650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:46.429528 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:46.432007 (+ 2479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:46.432013 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:46.432440 (+ 427us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:46.432628 (+ 188us) client_negotiation.cc:769] Sending connection context
0806 02:58:46.432780 (+ 152us) client_negotiation.cc:240] Negotiation successful
0806 02:58:46.433059 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1103,"mutex_wait_us":479,"spinlock_wait_cycles":52736,"thread_start_us":766,"threads_started":1}
I20260806 02:58:46.434188 1899 raft_consensus.cc:482] T 34dee7f4687243aaa359bda73a875b04 P f0ac4bacaea845ad987b992b68df2772 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:58:46.434293 1890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:46.421388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.193:45975 (local address 127.31.74.194:36655)
0806 02:58:46.421818 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:46.421825 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:46.422698 (+ 873us) server_negotiation.cc:407] Connection header received
0806 02:58:46.422979 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:46.422986 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:46.423077 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:46.423198 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:58:46.424255 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:46.425533 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:46.429890 (+ 4357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:46.431406 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:46.431700 (+ 294us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:46.431800 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:46.433689 (+ 1889us) server_negotiation.cc:299] Negotiation successful
0806 02:58:46.433940 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 02:58:46.434533 1899 raft_consensus.cc:504] T 34dee7f4687243aaa359bda73a875b04 P f0ac4bacaea845ad987b992b68df2772 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451b2658f7d946c8a4cc453e96d8613f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37965 } } peers { permanent_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 36655 } } peers { permanent_uuid: "f0ac4bacaea845ad987b992b68df2772" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 39003 } }
I20260806 02:58:46.435495 1680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34dee7f4687243aaa359bda73a875b04" candidate_uuid: "451b2658f7d946c8a4cc453e96d8613f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" is_pre_election: true
I20260806 02:58:46.435504 1822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34dee7f4687243aaa359bda73a875b04" candidate_uuid: "451b2658f7d946c8a4cc453e96d8613f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0ac4bacaea845ad987b992b68df2772" is_pre_election: true
I20260806 02:58:46.436102 1680 raft_consensus.cc:2454] T 34dee7f4687243aaa359bda73a875b04 P c2d58cba7d4847c4901dcd119b6bd6bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 451b2658f7d946c8a4cc453e96d8613f in term 0.
I20260806 02:58:46.436056 1822 raft_consensus.cc:2454] T 34dee7f4687243aaa359bda73a875b04 P f0ac4bacaea845ad987b992b68df2772 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 451b2658f7d946c8a4cc453e96d8613f in term 0.
I20260806 02:58:46.436659 1899 leader_election.cc:288] T 34dee7f4687243aaa359bda73a875b04 P f0ac4bacaea845ad987b992b68df2772 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 451b2658f7d946c8a4cc453e96d8613f (127.31.74.193:37965), c2d58cba7d4847c4901dcd119b6bd6bb (127.31.74.194:36655)
I20260806 02:58:46.437245 1467 leader_election.cc:302] T 34dee7f4687243aaa359bda73a875b04 P 451b2658f7d946c8a4cc453e96d8613f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 451b2658f7d946c8a4cc453e96d8613f, c2d58cba7d4847c4901dcd119b6bd6bb; no voters:
I20260806 02:58:46.438236 1898 raft_consensus.cc:2781] T 34dee7f4687243aaa359bda73a875b04 P 451b2658f7d946c8a4cc453e96d8613f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:58:46.438407 1898 raft_consensus.cc:482] T 34dee7f4687243aaa359bda73a875b04 P 451b2658f7d946c8a4cc453e96d8613f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:58:46.438534 1898 raft_consensus.cc:3037] T 34dee7f4687243aaa359bda73a875b04 P 451b2658f7d946c8a4cc453e96d8613f [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:58:46.442549 1898 raft_consensus.cc:504] T 34dee7f4687243aaa359bda73a875b04 P 451b2658f7d946c8a4cc453e96d8613f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451b2658f7d946c8a4cc453e96d8613f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37965 } } peers { permanent_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 36655 } } peers { permanent_uuid: "f0ac4bacaea845ad987b992b68df2772" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 39003 } }
I20260806 02:58:46.443919 1898 leader_election.cc:288] T 34dee7f4687243aaa359bda73a875b04 P 451b2658f7d946c8a4cc453e96d8613f [CANDIDATE]: Term 1 election: Requested vote from peers c2d58cba7d4847c4901dcd119b6bd6bb (127.31.74.194:36655), f0ac4bacaea845ad987b992b68df2772 (127.31.74.195:39003)
I20260806 02:58:46.444084 1822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34dee7f4687243aaa359bda73a875b04" candidate_uuid: "451b2658f7d946c8a4cc453e96d8613f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0ac4bacaea845ad987b992b68df2772"
I20260806 02:58:46.444375 1822 raft_consensus.cc:3037] T 34dee7f4687243aaa359bda73a875b04 P f0ac4bacaea845ad987b992b68df2772 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:58:46.444379 1680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34dee7f4687243aaa359bda73a875b04" candidate_uuid: "451b2658f7d946c8a4cc453e96d8613f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb"
I20260806 02:58:46.444736 1680 raft_consensus.cc:3037] T 34dee7f4687243aaa359bda73a875b04 P c2d58cba7d4847c4901dcd119b6bd6bb [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:58:46.446241 1761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:46.437028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:37965 (local address 127.31.74.195:33075)
0806 02:58:46.437810 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:46.437834 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 02:58:46.437995 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:58:46.439028 (+ 1033us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:58:46.439032 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:58:46.439070 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:58:46.439311 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:46.439321 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:46.440872 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:46.440878 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:46.443079 (+ 2201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:46.443105 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:46.445288 (+ 2183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:46.445295 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:46.445542 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:46.445584 (+ 42us) client_negotiation.cc:769] Sending connection context
0806 02:58:46.445690 (+ 106us) client_negotiation.cc:240] Negotiation successful
0806 02:58:46.445893 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650}
I20260806 02:58:46.447386 1887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:46.437007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.195:33075 (local address 127.31.74.193:37965)
0806 02:58:46.437294 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:46.437301 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:46.438573 (+ 1272us) server_negotiation.cc:407] Connection header received
0806 02:58:46.438709 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:46.438715 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:46.438804 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:46.438925 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:58:46.439472 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:46.440702 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:46.443514 (+ 2812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:46.444922 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:46.445097 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:46.445184 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:46.446671 (+ 1487us) server_negotiation.cc:299] Negotiation successful
0806 02:58:46.447065 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260806 02:58:46.448381 1903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:46.437512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.194:36655 (local address 127.31.74.195:57945)
0806 02:58:46.438271 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:46.438292 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 02:58:46.438423 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:58:46.439615 (+ 1192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:58:46.439621 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:58:46.439651 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:58:46.440079 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:46.440093 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:46.441987 (+ 1894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:46.441994 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:46.444610 (+ 2616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:46.444630 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:46.447295 (+ 2665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:46.447303 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:46.447570 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:46.447616 (+ 46us) client_negotiation.cc:769] Sending connection context
0806 02:58:46.447743 (+ 127us) client_negotiation.cc:240] Negotiation successful
0806 02:58:46.447899 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":433,"threads_started":1}
I20260806 02:58:46.448822 1537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34dee7f4687243aaa359bda73a875b04" candidate_uuid: "f0ac4bacaea845ad987b992b68df2772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "451b2658f7d946c8a4cc453e96d8613f" is_pre_election: true
I20260806 02:58:46.449419 1537 raft_consensus.cc:2379] T 34dee7f4687243aaa359bda73a875b04 P 451b2658f7d946c8a4cc453e96d8613f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f0ac4bacaea845ad987b992b68df2772 in current term 1: Already voted for candidate 451b2658f7d946c8a4cc453e96d8613f in this term.
I20260806 02:58:46.450874 1680 raft_consensus.cc:2454] T 34dee7f4687243aaa359bda73a875b04 P c2d58cba7d4847c4901dcd119b6bd6bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 451b2658f7d946c8a4cc453e96d8613f in term 1.
I20260806 02:58:46.451869 1890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:46.437574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.195:57945 (local address 127.31.74.194:36655)
0806 02:58:46.439285 (+ 1711us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:46.439294 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:46.439318 (+ 24us) server_negotiation.cc:407] Connection header received
0806 02:58:46.439390 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:46.439395 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:46.439477 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:46.439580 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:58:46.440263 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:46.441734 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:46.444855 (+ 3121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:46.446543 (+ 1688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:46.446955 (+ 412us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:46.447050 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:46.447776 (+ 726us) server_negotiation.cc:299] Negotiation successful
0806 02:58:46.447935 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1563}
I20260806 02:58:46.451859 1467 leader_election.cc:302] T 34dee7f4687243aaa359bda73a875b04 P 451b2658f7d946c8a4cc453e96d8613f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 451b2658f7d946c8a4cc453e96d8613f, c2d58cba7d4847c4901dcd119b6bd6bb; no voters:
I20260806 02:58:46.452451 1680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34dee7f4687243aaa359bda73a875b04" candidate_uuid: "f0ac4bacaea845ad987b992b68df2772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" is_pre_election: true
I20260806 02:58:46.452636 1822 raft_consensus.cc:2454] T 34dee7f4687243aaa359bda73a875b04 P f0ac4bacaea845ad987b992b68df2772 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 451b2658f7d946c8a4cc453e96d8613f in term 1.
I20260806 02:58:46.452965 1680 raft_consensus.cc:2379] T 34dee7f4687243aaa359bda73a875b04 P c2d58cba7d4847c4901dcd119b6bd6bb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f0ac4bacaea845ad987b992b68df2772 in current term 1: Already voted for candidate 451b2658f7d946c8a4cc453e96d8613f in this term.
I20260806 02:58:46.453262 1898 raft_consensus.cc:2781] T 34dee7f4687243aaa359bda73a875b04 P 451b2658f7d946c8a4cc453e96d8613f [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:58:46.453727 1752 leader_election.cc:302] T 34dee7f4687243aaa359bda73a875b04 P f0ac4bacaea845ad987b992b68df2772 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0ac4bacaea845ad987b992b68df2772; no voters: 451b2658f7d946c8a4cc453e96d8613f, c2d58cba7d4847c4901dcd119b6bd6bb
I20260806 02:58:46.453987 1898 raft_consensus.cc:686] T 34dee7f4687243aaa359bda73a875b04 P 451b2658f7d946c8a4cc453e96d8613f [term 1 LEADER]: Becoming Leader. State: Replica: 451b2658f7d946c8a4cc453e96d8613f, State: Running, Role: LEADER
I20260806 02:58:46.454420 1899 raft_consensus.cc:2726] T 34dee7f4687243aaa359bda73a875b04 P f0ac4bacaea845ad987b992b68df2772 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 02:58:46.454955 1898 consensus_queue.cc:237] T 34dee7f4687243aaa359bda73a875b04 P 451b2658f7d946c8a4cc453e96d8613f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451b2658f7d946c8a4cc453e96d8613f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37965 } } peers { permanent_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 36655 } } peers { permanent_uuid: "f0ac4bacaea845ad987b992b68df2772" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 39003 } }
I20260806 02:58:46.460702 1378 catalog_manager.cc:5495] T 34dee7f4687243aaa359bda73a875b04 P 451b2658f7d946c8a4cc453e96d8613f reported cstate change: term changed from 0 to 1, leader changed from <none> to 451b2658f7d946c8a4cc453e96d8613f (127.31.74.193). New cstate: current_term: 1 leader_uuid: "451b2658f7d946c8a4cc453e96d8613f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451b2658f7d946c8a4cc453e96d8613f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 36655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0ac4bacaea845ad987b992b68df2772" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 39003 } health_report { overall_health: UNKNOWN } } }
Aug 06 02:58:46 dist-test-slave-dhph krb5kdc[1333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985123, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.254@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 02:58:46.560804 1911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:46.542186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:42053 (local address 127.0.0.1:36850)
0806 02:58:46.543271 (+ 1085us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:46.543290 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 02:58:46.543456 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:58:46.544261 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:58:46.544266 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:58:46.545108 (+ 842us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:58:46.545401 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:46.545411 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:46.546660 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:46.546666 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:46.548058 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:46.548073 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:46.548971 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:46.548977 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:46.549114 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:46.550042 (+ 928us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:58:46.550061 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:58:46.553498 (+ 3437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:58:46.557281 (+ 3783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:58:46.557302 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:58:46.557309 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:58:46.558072 (+ 763us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:58:46.558798 (+ 726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:58:46.558802 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:58:46.558805 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:58:46.558974 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:58:46.559684 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:58:46.559691 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:58:46.559892 (+ 201us) client_negotiation.cc:769] Sending connection context
0806 02:58:46.560073 (+ 181us) client_negotiation.cc:240] Negotiation successful
0806 02:58:46.560359 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":926,"thread_start_us":729,"threads_started":1}
I20260806 02:58:46.561008 1759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:46.542139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36850 (local address 127.31.74.254:42053)
0806 02:58:46.542390 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:46.542399 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:46.543420 (+ 1021us) server_negotiation.cc:407] Connection header received
0806 02:58:46.543739 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:46.543746 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:46.543856 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:46.544192 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:46.545589 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:46.546437 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:46.548234 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:46.548722 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:46.548903 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:46.553760 (+ 4857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:58:46.553766 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:58:46.553770 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:58:46.553822 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:58:46.557053 (+ 3231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:46.558271 (+ 1218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:46.558276 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:46.558278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:46.558417 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:46.559163 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:46.559169 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:46.559173 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:46.559494 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:58:46.559690 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:46.560218 (+ 528us) server_negotiation.cc:299] Negotiation successful
0806 02:58:46.560542 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":102,"server-negotiator.queue_time_us":100}
I20260806 02:58:46.568528 1378 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.31.74.254@KRBTEST.COM'} at 127.0.0.1:36850:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260806 02:58:46.569736 1378 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:58:46.577900 1517 tablet_service.cc:1467] Processing CreateTablet for tablet 19dd246d3b934aa5885b616b8efec58d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=303aa5cc03b94415bc94cbcca62970ac]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 02:58:46.578270 1802 tablet_service.cc:1467] Processing CreateTablet for tablet 19dd246d3b934aa5885b616b8efec58d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=303aa5cc03b94415bc94cbcca62970ac]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 02:58:46.578270 1659 tablet_service.cc:1467] Processing CreateTablet for tablet 19dd246d3b934aa5885b616b8efec58d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=303aa5cc03b94415bc94cbcca62970ac]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 02:58:46.578985 1517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19dd246d3b934aa5885b616b8efec58d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:58:46.579391 1802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19dd246d3b934aa5885b616b8efec58d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:58:46.579396 1659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19dd246d3b934aa5885b616b8efec58d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:58:46.586299 1893 tablet_bootstrap.cc:492] T 19dd246d3b934aa5885b616b8efec58d P 451b2658f7d946c8a4cc453e96d8613f: Bootstrap starting.
I20260806 02:58:46.587136 1894 tablet_bootstrap.cc:492] T 19dd246d3b934aa5885b616b8efec58d P c2d58cba7d4847c4901dcd119b6bd6bb: Bootstrap starting.
I20260806 02:58:46.587136 1892 tablet_bootstrap.cc:492] T 19dd246d3b934aa5885b616b8efec58d P f0ac4bacaea845ad987b992b68df2772: Bootstrap starting.
I20260806 02:58:46.588403 1893 tablet_bootstrap.cc:654] T 19dd246d3b934aa5885b616b8efec58d P 451b2658f7d946c8a4cc453e96d8613f: Neither blocks nor log segments found. Creating new log.
I20260806 02:58:46.589264 1892 tablet_bootstrap.cc:654] T 19dd246d3b934aa5885b616b8efec58d P f0ac4bacaea845ad987b992b68df2772: Neither blocks nor log segments found. Creating new log.
I20260806 02:58:46.589283 1894 tablet_bootstrap.cc:654] T 19dd246d3b934aa5885b616b8efec58d P c2d58cba7d4847c4901dcd119b6bd6bb: Neither blocks nor log segments found. Creating new log.
I20260806 02:58:46.593357 1894 tablet_bootstrap.cc:492] T 19dd246d3b934aa5885b616b8efec58d P c2d58cba7d4847c4901dcd119b6bd6bb: No bootstrap required, opened a new log
I20260806 02:58:46.593392 1892 tablet_bootstrap.cc:492] T 19dd246d3b934aa5885b616b8efec58d P f0ac4bacaea845ad987b992b68df2772: No bootstrap required, opened a new log
I20260806 02:58:46.593400 1893 tablet_bootstrap.cc:492] T 19dd246d3b934aa5885b616b8efec58d P 451b2658f7d946c8a4cc453e96d8613f: No bootstrap required, opened a new log
I20260806 02:58:46.593554 1892 ts_tablet_manager.cc:1397] T 19dd246d3b934aa5885b616b8efec58d P f0ac4bacaea845ad987b992b68df2772: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.004s
I20260806 02:58:46.593554 1894 ts_tablet_manager.cc:1397] T 19dd246d3b934aa5885b616b8efec58d P c2d58cba7d4847c4901dcd119b6bd6bb: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260806 02:58:46.593591 1893 ts_tablet_manager.cc:1397] T 19dd246d3b934aa5885b616b8efec58d P 451b2658f7d946c8a4cc453e96d8613f: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260806 02:58:46.594410 1892 raft_consensus.cc:348] T 19dd246d3b934aa5885b616b8efec58d P f0ac4bacaea845ad987b992b68df2772 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451b2658f7d946c8a4cc453e96d8613f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37965 } } peers { permanent_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 36655 } } peers { permanent_uuid: "f0ac4bacaea845ad987b992b68df2772" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 39003 } }
I20260806 02:58:46.594412 1894 raft_consensus.cc:348] T 19dd246d3b934aa5885b616b8efec58d P c2d58cba7d4847c4901dcd119b6bd6bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451b2658f7d946c8a4cc453e96d8613f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37965 } } peers { permanent_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 36655 } } peers { permanent_uuid: "f0ac4bacaea845ad987b992b68df2772" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 39003 } }
I20260806 02:58:46.594743 1892 raft_consensus.cc:374] T 19dd246d3b934aa5885b616b8efec58d P f0ac4bacaea845ad987b992b68df2772 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:58:46.594743 1894 raft_consensus.cc:374] T 19dd246d3b934aa5885b616b8efec58d P c2d58cba7d4847c4901dcd119b6bd6bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:58:46.594853 1892 raft_consensus.cc:729] T 19dd246d3b934aa5885b616b8efec58d P f0ac4bacaea845ad987b992b68df2772 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0ac4bacaea845ad987b992b68df2772, State: Initialized, Role: FOLLOWER
I20260806 02:58:46.594853 1894 raft_consensus.cc:729] T 19dd246d3b934aa5885b616b8efec58d P c2d58cba7d4847c4901dcd119b6bd6bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2d58cba7d4847c4901dcd119b6bd6bb, State: Initialized, Role: FOLLOWER
I20260806 02:58:46.594923 1893 raft_consensus.cc:348] T 19dd246d3b934aa5885b616b8efec58d P 451b2658f7d946c8a4cc453e96d8613f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451b2658f7d946c8a4cc453e96d8613f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37965 } } peers { permanent_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 36655 } } peers { permanent_uuid: "f0ac4bacaea845ad987b992b68df2772" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 39003 } }
I20260806 02:58:46.595192 1893 raft_consensus.cc:374] T 19dd246d3b934aa5885b616b8efec58d P 451b2658f7d946c8a4cc453e96d8613f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:58:46.595110 1894 consensus_queue.cc:260] T 19dd246d3b934aa5885b616b8efec58d P c2d58cba7d4847c4901dcd119b6bd6bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451b2658f7d946c8a4cc453e96d8613f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37965 } } peers { permanent_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 36655 } } peers { permanent_uuid: "f0ac4bacaea845ad987b992b68df2772" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 39003 } }
I20260806 02:58:46.595132 1892 consensus_queue.cc:260] T 19dd246d3b934aa5885b616b8efec58d P f0ac4bacaea845ad987b992b68df2772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451b2658f7d946c8a4cc453e96d8613f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37965 } } peers { permanent_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 36655 } } peers { permanent_uuid: "f0ac4bacaea845ad987b992b68df2772" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 39003 } }
I20260806 02:58:46.595302 1893 raft_consensus.cc:729] T 19dd246d3b934aa5885b616b8efec58d P 451b2658f7d946c8a4cc453e96d8613f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 451b2658f7d946c8a4cc453e96d8613f, State: Initialized, Role: FOLLOWER
I20260806 02:58:46.595742 1894 ts_tablet_manager.cc:1428] T 19dd246d3b934aa5885b616b8efec58d P c2d58cba7d4847c4901dcd119b6bd6bb: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260806 02:58:46.595556 1893 consensus_queue.cc:260] T 19dd246d3b934aa5885b616b8efec58d P 451b2658f7d946c8a4cc453e96d8613f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451b2658f7d946c8a4cc453e96d8613f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37965 } } peers { permanent_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 36655 } } peers { permanent_uuid: "f0ac4bacaea845ad987b992b68df2772" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 39003 } }
I20260806 02:58:46.596386 1893 ts_tablet_manager.cc:1428] T 19dd246d3b934aa5885b616b8efec58d P 451b2658f7d946c8a4cc453e96d8613f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260806 02:58:46.596114 1899 tablet_replica.cc:439] T 19dd246d3b934aa5885b616b8efec58d P f0ac4bacaea845ad987b992b68df2772: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451b2658f7d946c8a4cc453e96d8613f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37965 } } peers { permanent_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 36655 } } peers { permanent_uuid: "f0ac4bacaea845ad987b992b68df2772" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 39003 } } }
I20260806 02:58:46.597313 1900 tablet_replica.cc:439] T 19dd246d3b934aa5885b616b8efec58d P c2d58cba7d4847c4901dcd119b6bd6bb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451b2658f7d946c8a4cc453e96d8613f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37965 } } peers { permanent_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 36655 } } peers { permanent_uuid: "f0ac4bacaea845ad987b992b68df2772" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 39003 } } }
I20260806 02:58:46.597702 1900 tablet_replica.cc:442] T 19dd246d3b934aa5885b616b8efec58d P c2d58cba7d4847c4901dcd119b6bd6bb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:58:46.597585 1898 tablet_replica.cc:439] T 19dd246d3b934aa5885b616b8efec58d P 451b2658f7d946c8a4cc453e96d8613f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451b2658f7d946c8a4cc453e96d8613f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37965 } } peers { permanent_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 36655 } } peers { permanent_uuid: "f0ac4bacaea845ad987b992b68df2772" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 39003 } } }
I20260806 02:58:46.597961 1898 tablet_replica.cc:442] T 19dd246d3b934aa5885b616b8efec58d P 451b2658f7d946c8a4cc453e96d8613f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:58:46.596983 1899 tablet_replica.cc:442] T 19dd246d3b934aa5885b616b8efec58d P f0ac4bacaea845ad987b992b68df2772: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:58:46.599941 1892 ts_tablet_manager.cc:1428] T 19dd246d3b934aa5885b616b8efec58d P f0ac4bacaea845ad987b992b68df2772: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260806 02:58:46.662982 1900 raft_consensus.cc:482] T 19dd246d3b934aa5885b616b8efec58d P c2d58cba7d4847c4901dcd119b6bd6bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:58:46.663460 1900 raft_consensus.cc:504] T 19dd246d3b934aa5885b616b8efec58d P c2d58cba7d4847c4901dcd119b6bd6bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451b2658f7d946c8a4cc453e96d8613f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37965 } } peers { permanent_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 36655 } } peers { permanent_uuid: "f0ac4bacaea845ad987b992b68df2772" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 39003 } }
I20260806 02:58:46.665556 1900 leader_election.cc:288] T 19dd246d3b934aa5885b616b8efec58d P c2d58cba7d4847c4901dcd119b6bd6bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 451b2658f7d946c8a4cc453e96d8613f (127.31.74.193:37965), f0ac4bacaea845ad987b992b68df2772 (127.31.74.195:39003)
I20260806 02:58:46.678889 1887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:46.666449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.194:49873 (local address 127.31.74.193:37965)
0806 02:58:46.666823 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:46.666834 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:46.668502 (+ 1668us) server_negotiation.cc:407] Connection header received
0806 02:58:46.668699 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:46.668703 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:46.668773 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:46.668880 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:58:46.670334 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:46.671488 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:46.674894 (+ 3406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:46.676979 (+ 2085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:46.677267 (+ 288us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:46.677369 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:46.678158 (+ 789us) server_negotiation.cc:299] Negotiation successful
0806 02:58:46.678512 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260806 02:58:46.678885 1891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:46.666245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.194:50001 (local address 127.31.74.195:39003)
0806 02:58:46.666778 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:46.666787 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:46.668150 (+ 1363us) server_negotiation.cc:407] Connection header received
0806 02:58:46.668382 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:46.668390 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:46.668486 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:46.668592 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:58:46.670323 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:46.671982 (+ 1659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:46.675150 (+ 3168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:46.677029 (+ 1879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:46.677267 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:46.677369 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:46.678158 (+ 789us) server_negotiation.cc:299] Negotiation successful
0806 02:58:46.678467 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351}
I20260806 02:58:46.678869 1916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:46.666317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:37965 (local address 127.31.74.194:49873)
0806 02:58:46.668221 (+ 1904us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:46.668242 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 02:58:46.668525 (+ 283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:58:46.669054 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:58:46.669062 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:58:46.669288 (+ 226us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:58:46.670009 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:46.670026 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:46.671631 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:46.671638 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:46.674546 (+ 2908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:46.674570 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:46.677433 (+ 2863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:46.677438 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:46.677769 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:46.677886 (+ 117us) client_negotiation.cc:769] Sending connection context
0806 02:58:46.678019 (+ 133us) client_negotiation.cc:240] Negotiation successful
0806 02:58:46.678301 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1511,"mutex_wait_us":335,"spinlock_wait_cycles":16768,"thread_start_us":1134,"threads_started":1}
I20260806 02:58:46.678908 1915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:46.666262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.195:39003 (local address 127.31.74.194:50001)
0806 02:58:46.667869 (+ 1607us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:46.667902 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0806 02:58:46.668101 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:58:46.669054 (+ 953us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:58:46.669062 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:58:46.669150 (+ 88us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:58:46.670012 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:46.670026 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:46.672162 (+ 2136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:46.672168 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:46.674918 (+ 2750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:46.674930 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:46.677425 (+ 2495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:46.677433 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:46.677772 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:46.677886 (+ 114us) client_negotiation.cc:769] Sending connection context
0806 02:58:46.678020 (+ 134us) client_negotiation.cc:240] Negotiation successful
0806 02:58:46.678301 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1419,"mutex_wait_us":136,"spinlock_wait_cycles":53760,"thread_start_us":1102,"threads_started":1}
I20260806 02:58:46.679688 1822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19dd246d3b934aa5885b616b8efec58d" candidate_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0ac4bacaea845ad987b992b68df2772" is_pre_election: true
I20260806 02:58:46.679900 1537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19dd246d3b934aa5885b616b8efec58d" candidate_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "451b2658f7d946c8a4cc453e96d8613f" is_pre_election: true
I20260806 02:58:46.680115 1822 raft_consensus.cc:2454] T 19dd246d3b934aa5885b616b8efec58d P f0ac4bacaea845ad987b992b68df2772 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2d58cba7d4847c4901dcd119b6bd6bb in term 0.
I20260806 02:58:46.680171 1537 raft_consensus.cc:2454] T 19dd246d3b934aa5885b616b8efec58d P 451b2658f7d946c8a4cc453e96d8613f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2d58cba7d4847c4901dcd119b6bd6bb in term 0.
I20260806 02:58:46.680727 1610 leader_election.cc:302] T 19dd246d3b934aa5885b616b8efec58d P c2d58cba7d4847c4901dcd119b6bd6bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2d58cba7d4847c4901dcd119b6bd6bb, f0ac4bacaea845ad987b992b68df2772; no voters:
I20260806 02:58:46.681270 1900 raft_consensus.cc:2781] T 19dd246d3b934aa5885b616b8efec58d P c2d58cba7d4847c4901dcd119b6bd6bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:58:46.681514 1900 raft_consensus.cc:482] T 19dd246d3b934aa5885b616b8efec58d P c2d58cba7d4847c4901dcd119b6bd6bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:58:46.681675 1900 raft_consensus.cc:3037] T 19dd246d3b934aa5885b616b8efec58d P c2d58cba7d4847c4901dcd119b6bd6bb [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:58:46.685225 1900 raft_consensus.cc:504] T 19dd246d3b934aa5885b616b8efec58d P c2d58cba7d4847c4901dcd119b6bd6bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451b2658f7d946c8a4cc453e96d8613f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37965 } } peers { permanent_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 36655 } } peers { permanent_uuid: "f0ac4bacaea845ad987b992b68df2772" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 39003 } }
I20260806 02:58:46.686050 1900 leader_election.cc:288] T 19dd246d3b934aa5885b616b8efec58d P c2d58cba7d4847c4901dcd119b6bd6bb [CANDIDATE]: Term 1 election: Requested vote from peers 451b2658f7d946c8a4cc453e96d8613f (127.31.74.193:37965), f0ac4bacaea845ad987b992b68df2772 (127.31.74.195:39003)
I20260806 02:58:46.686707 1537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19dd246d3b934aa5885b616b8efec58d" candidate_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "451b2658f7d946c8a4cc453e96d8613f"
I20260806 02:58:46.686810 1822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19dd246d3b934aa5885b616b8efec58d" candidate_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0ac4bacaea845ad987b992b68df2772"
I20260806 02:58:46.687001 1537 raft_consensus.cc:3037] T 19dd246d3b934aa5885b616b8efec58d P 451b2658f7d946c8a4cc453e96d8613f [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:58:46.687073 1822 raft_consensus.cc:3037] T 19dd246d3b934aa5885b616b8efec58d P f0ac4bacaea845ad987b992b68df2772 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:58:46.691169 1537 raft_consensus.cc:2454] T 19dd246d3b934aa5885b616b8efec58d P 451b2658f7d946c8a4cc453e96d8613f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2d58cba7d4847c4901dcd119b6bd6bb in term 1.
I20260806 02:58:46.691715 1822 raft_consensus.cc:2454] T 19dd246d3b934aa5885b616b8efec58d P f0ac4bacaea845ad987b992b68df2772 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2d58cba7d4847c4901dcd119b6bd6bb in term 1.
I20260806 02:58:46.691810 1611 leader_election.cc:302] T 19dd246d3b934aa5885b616b8efec58d P c2d58cba7d4847c4901dcd119b6bd6bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 451b2658f7d946c8a4cc453e96d8613f, c2d58cba7d4847c4901dcd119b6bd6bb; no voters:
I20260806 02:58:46.692270 1900 raft_consensus.cc:2781] T 19dd246d3b934aa5885b616b8efec58d P c2d58cba7d4847c4901dcd119b6bd6bb [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:58:46.692955 1900 raft_consensus.cc:686] T 19dd246d3b934aa5885b616b8efec58d P c2d58cba7d4847c4901dcd119b6bd6bb [term 1 LEADER]: Becoming Leader. State: Replica: c2d58cba7d4847c4901dcd119b6bd6bb, State: Running, Role: LEADER
I20260806 02:58:46.693775 1900 consensus_queue.cc:237] T 19dd246d3b934aa5885b616b8efec58d P c2d58cba7d4847c4901dcd119b6bd6bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451b2658f7d946c8a4cc453e96d8613f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37965 } } peers { permanent_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 36655 } } peers { permanent_uuid: "f0ac4bacaea845ad987b992b68df2772" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 39003 } }
I20260806 02:58:46.697880 1917 tablet_replica.cc:439] T 19dd246d3b934aa5885b616b8efec58d P c2d58cba7d4847c4901dcd119b6bd6bb: TxnStatusTablet state changed. Reason: New leader c2d58cba7d4847c4901dcd119b6bd6bb. Latest consensus state: current_term: 1 leader_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451b2658f7d946c8a4cc453e96d8613f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37965 } } peers { permanent_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 36655 } } peers { permanent_uuid: "f0ac4bacaea845ad987b992b68df2772" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 39003 } } }
I20260806 02:58:46.698531 1917 tablet_replica.cc:442] T 19dd246d3b934aa5885b616b8efec58d P c2d58cba7d4847c4901dcd119b6bd6bb: This TxnStatusTablet replica's current role is: LEADER
I20260806 02:58:46.699445 1919 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260806 02:58:46.699788 1919 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260806 02:58:46.700055 1377 catalog_manager.cc:5495] T 19dd246d3b934aa5885b616b8efec58d P c2d58cba7d4847c4901dcd119b6bd6bb reported cstate change: term changed from 0 to 1, leader changed from <none> to c2d58cba7d4847c4901dcd119b6bd6bb (127.31.74.194). New cstate: current_term: 1 leader_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451b2658f7d946c8a4cc453e96d8613f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 36655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0ac4bacaea845ad987b992b68df2772" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 39003 } health_report { overall_health: UNKNOWN } } }
I20260806 02:58:46.735669 1890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:46.727043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37256 (local address 127.31.74.194:36655)
0806 02:58:46.727258 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:46.727264 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:46.727286 (+ 22us) server_negotiation.cc:407] Connection header received
0806 02:58:46.727666 (+ 380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:46.727673 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:46.727814 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:46.727995 (+ 181us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:58:46.728503 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:46.729843 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:46.732184 (+ 2341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:46.733026 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:46.733194 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:46.733846 (+ 652us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:58:46.734917 (+ 1071us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:58:46.735083 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:46.735218 (+ 135us) server_negotiation.cc:299] Negotiation successful
0806 02:58:46.735338 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260806 02:58:46.735726 1911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:46.726826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.194:36655 (local address 127.0.0.1:37256)
0806 02:58:46.727096 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:46.727120 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 02:58:46.727351 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:58:46.728066 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:58:46.728072 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:58:46.728087 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 02:58:46.728336 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:46.728347 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:46.730047 (+ 1700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:46.730054 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:46.731993 (+ 1939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:46.732010 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:46.733286 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:46.733293 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:46.733496 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:46.733585 (+ 89us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 02:58:46.735049 (+ 1464us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 02:58:46.735078 (+ 29us) client_negotiation.cc:769] Sending connection context
0806 02:58:46.735186 (+ 108us) client_negotiation.cc:240] Negotiation successful
0806 02:58:46.735376 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260806 02:58:46.748509 1822 raft_consensus.cc:1261] T 19dd246d3b934aa5885b616b8efec58d P f0ac4bacaea845ad987b992b68df2772 [term 1 FOLLOWER]: Refusing update from remote peer c2d58cba7d4847c4901dcd119b6bd6bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:58:46.748584 1537 raft_consensus.cc:1261] T 19dd246d3b934aa5885b616b8efec58d P 451b2658f7d946c8a4cc453e96d8613f [term 1 FOLLOWER]: Refusing update from remote peer c2d58cba7d4847c4901dcd119b6bd6bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:58:46.749660 1900 consensus_queue.cc:1035] T 19dd246d3b934aa5885b616b8efec58d P c2d58cba7d4847c4901dcd119b6bd6bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0ac4bacaea845ad987b992b68df2772" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 39003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:58:46.750238 1917 consensus_queue.cc:1035] T 19dd246d3b934aa5885b616b8efec58d P c2d58cba7d4847c4901dcd119b6bd6bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "451b2658f7d946c8a4cc453e96d8613f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:58:46.760135 1898 tablet_replica.cc:439] T 19dd246d3b934aa5885b616b8efec58d P 451b2658f7d946c8a4cc453e96d8613f: TxnStatusTablet state changed. Reason: New leader c2d58cba7d4847c4901dcd119b6bd6bb. Latest consensus state: current_term: 1 leader_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451b2658f7d946c8a4cc453e96d8613f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37965 } } peers { permanent_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 36655 } } peers { permanent_uuid: "f0ac4bacaea845ad987b992b68df2772" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 39003 } } }
I20260806 02:58:46.760705 1898 tablet_replica.cc:442] T 19dd246d3b934aa5885b616b8efec58d P 451b2658f7d946c8a4cc453e96d8613f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:58:46.763779 1922 mvcc.cc:205] Tried to move back new op lower bound from 7315395079148617728 to 7315395078942380032. Current Snapshot: MvccSnapshot[applied={T|T < 7315395079148617728}]
I20260806 02:58:46.767417 1899 tablet_replica.cc:439] T 19dd246d3b934aa5885b616b8efec58d P f0ac4bacaea845ad987b992b68df2772: TxnStatusTablet state changed. Reason: New leader c2d58cba7d4847c4901dcd119b6bd6bb. Latest consensus state: current_term: 1 leader_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451b2658f7d946c8a4cc453e96d8613f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37965 } } peers { permanent_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 36655 } } peers { permanent_uuid: "f0ac4bacaea845ad987b992b68df2772" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 39003 } } }
I20260806 02:58:46.767824 1899 tablet_replica.cc:442] T 19dd246d3b934aa5885b616b8efec58d P f0ac4bacaea845ad987b992b68df2772: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:58:46.774048 1900 tablet_replica.cc:439] T 19dd246d3b934aa5885b616b8efec58d P c2d58cba7d4847c4901dcd119b6bd6bb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451b2658f7d946c8a4cc453e96d8613f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37965 } } peers { permanent_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 36655 } } peers { permanent_uuid: "f0ac4bacaea845ad987b992b68df2772" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 39003 } } }
I20260806 02:58:46.774420 1900 tablet_replica.cc:442] T 19dd246d3b934aa5885b616b8efec58d P c2d58cba7d4847c4901dcd119b6bd6bb: This TxnStatusTablet replica's current role is: LEADER
I20260806 02:58:46.774921 1898 tablet_replica.cc:439] T 19dd246d3b934aa5885b616b8efec58d P 451b2658f7d946c8a4cc453e96d8613f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451b2658f7d946c8a4cc453e96d8613f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37965 } } peers { permanent_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 36655 } } peers { permanent_uuid: "f0ac4bacaea845ad987b992b68df2772" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 39003 } } }
I20260806 02:58:46.775262 1898 tablet_replica.cc:442] T 19dd246d3b934aa5885b616b8efec58d P 451b2658f7d946c8a4cc453e96d8613f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:58:46.776729 1900 tablet_replica.cc:439] T 19dd246d3b934aa5885b616b8efec58d P c2d58cba7d4847c4901dcd119b6bd6bb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451b2658f7d946c8a4cc453e96d8613f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37965 } } peers { permanent_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 36655 } } peers { permanent_uuid: "f0ac4bacaea845ad987b992b68df2772" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 39003 } } }
I20260806 02:58:46.777046 1900 tablet_replica.cc:442] T 19dd246d3b934aa5885b616b8efec58d P c2d58cba7d4847c4901dcd119b6bd6bb: This TxnStatusTablet replica's current role is: LEADER
I20260806 02:58:46.784381 1899 tablet_replica.cc:439] T 19dd246d3b934aa5885b616b8efec58d P f0ac4bacaea845ad987b992b68df2772: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451b2658f7d946c8a4cc453e96d8613f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37965 } } peers { permanent_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 36655 } } peers { permanent_uuid: "f0ac4bacaea845ad987b992b68df2772" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 39003 } } }
I20260806 02:58:46.784721 1899 tablet_replica.cc:442] T 19dd246d3b934aa5885b616b8efec58d P f0ac4bacaea845ad987b992b68df2772: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:58:46.795233 1887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:46.787993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36952 (local address 127.31.74.193:37965)
0806 02:58:46.788346 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:46.788357 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:46.789196 (+ 839us) server_negotiation.cc:407] Connection header received
0806 02:58:46.789490 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:46.789495 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:46.789643 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:46.789822 (+ 179us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:58:46.790616 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:46.791427 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:46.792853 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:46.793280 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:46.793428 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:46.793885 (+ 457us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:58:46.794460 (+ 575us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:58:46.794641 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:46.794762 (+ 121us) server_negotiation.cc:299] Negotiation successful
0806 02:58:46.794852 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260806 02:58:46.823287 1890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:46.808407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.193:60273 (local address 127.31.74.194:36655)
0806 02:58:46.808673 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:46.808681 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:46.816195 (+ 7514us) server_negotiation.cc:407] Connection header received
0806 02:58:46.816332 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:46.816338 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:46.816458 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:46.816559 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:58:46.817387 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:46.818201 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:46.820624 (+ 2423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:46.821299 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:46.821428 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:46.822287 (+ 859us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:58:46.822434 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:58:46.822543 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:46.822817 (+ 274us) server_negotiation.cc:299] Negotiation successful
0806 02:58:46.822994 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260806 02:58:46.823824 1934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:46.808203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.194:36655 (local address 127.31.74.193:60273)
0806 02:58:46.815901 (+ 7698us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:46.815930 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0806 02:58:46.816157 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:58:46.816602 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:58:46.816607 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:58:46.816623 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 02:58:46.817042 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:46.817056 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:46.818367 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:46.818373 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:46.820436 (+ 2063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:46.820457 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:46.821610 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:46.821623 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:46.821825 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:46.821922 (+ 97us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 02:58:46.822568 (+ 646us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 02:58:46.822621 (+ 53us) client_negotiation.cc:769] Sending connection context
0806 02:58:46.822715 (+ 94us) client_negotiation.cc:240] Negotiation successful
0806 02:58:46.823168 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7402,"thread_start_us":7195,"threads_started":1}
I20260806 02:58:46.838181 1822 raft_consensus.cc:1261] T 34dee7f4687243aaa359bda73a875b04 P f0ac4bacaea845ad987b992b68df2772 [term 1 FOLLOWER]: Refusing update from remote peer 451b2658f7d946c8a4cc453e96d8613f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:58:46.838513 1680 raft_consensus.cc:1261] T 34dee7f4687243aaa359bda73a875b04 P c2d58cba7d4847c4901dcd119b6bd6bb [term 1 FOLLOWER]: Refusing update from remote peer 451b2658f7d946c8a4cc453e96d8613f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:58:46.839057 1904 consensus_queue.cc:1035] T 34dee7f4687243aaa359bda73a875b04 P 451b2658f7d946c8a4cc453e96d8613f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0ac4bacaea845ad987b992b68df2772" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 39003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:58:46.839670 1898 consensus_queue.cc:1035] T 34dee7f4687243aaa359bda73a875b04 P 451b2658f7d946c8a4cc453e96d8613f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 36655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:58:46.891328 1887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:46.883019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.194:35367 (local address 127.31.74.193:37965)
0806 02:58:46.883307 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:46.883316 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:46.884366 (+ 1050us) server_negotiation.cc:407] Connection header received
0806 02:58:46.884608 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:46.884614 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:46.884713 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:46.884917 (+ 204us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:58:46.885582 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:46.886502 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:46.888558 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:46.889624 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:46.889823 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:46.890344 (+ 521us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:58:46.890508 (+ 164us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:58:46.890656 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:46.890809 (+ 153us) server_negotiation.cc:299] Negotiation successful
0806 02:58:46.890901 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260806 02:58:46.891466 1941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:46.883253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:37965 (local address 127.31.74.194:35367)
0806 02:58:46.884225 (+ 972us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:46.884249 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 02:58:46.884444 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:58:46.884950 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:58:46.884954 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:58:46.884969 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 02:58:46.885413 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:46.885424 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:46.886714 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:46.886722 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:46.888372 (+ 1650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:46.888390 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:46.889882 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:46.889896 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:46.890053 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:46.890136 (+ 83us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 02:58:46.890643 (+ 507us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 02:58:46.890657 (+ 14us) client_negotiation.cc:769] Sending connection context
0806 02:58:46.890769 (+ 112us) client_negotiation.cc:240] Negotiation successful
0806 02:58:46.890934 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":721,"thread_start_us":533,"threads_started":1}
I20260806 02:58:46.958632 1378 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36838:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:58:46.959144 1378 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36838:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:58:46.966773 1378 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 24ba9714cb1a494d983288e0ab1b4283: Sending DeleteTablet for 3 replicas of tablet 34dee7f4687243aaa359bda73a875b04
I20260806 02:58:46.969096 1517 tablet_service.cc:1514] Processing DeleteTablet for tablet 34dee7f4687243aaa359bda73a875b04 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:58:46 UTC) from {username='kudu', principal='kudu/127.31.74.254@KRBTEST.COM'} at 127.0.0.1:36948
I20260806 02:58:46.969304 1659 tablet_service.cc:1514] Processing DeleteTablet for tablet 34dee7f4687243aaa359bda73a875b04 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:58:46 UTC) from {username='kudu', principal='kudu/127.31.74.254@KRBTEST.COM'} at 127.0.0.1:37248
I20260806 02:58:46.971408 1802 tablet_service.cc:1514] Processing DeleteTablet for tablet 34dee7f4687243aaa359bda73a875b04 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:58:46 UTC) from {username='kudu', principal='kudu/127.31.74.254@KRBTEST.COM'} at 127.0.0.1:60902
I20260806 02:58:46.975593 1378 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36838:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:58:46.977612 1378 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:58:46.981487 1946 tablet_replica.cc:330] T 34dee7f4687243aaa359bda73a875b04 P c2d58cba7d4847c4901dcd119b6bd6bb: stopping tablet replica
I20260806 02:58:46.991374 1947 tablet_replica.cc:330] T 34dee7f4687243aaa359bda73a875b04 P f0ac4bacaea845ad987b992b68df2772: stopping tablet replica
I20260806 02:58:46.991945 1947 raft_consensus.cc:2229] T 34dee7f4687243aaa359bda73a875b04 P f0ac4bacaea845ad987b992b68df2772 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:58:46.992308 1947 raft_consensus.cc:2258] T 34dee7f4687243aaa359bda73a875b04 P f0ac4bacaea845ad987b992b68df2772 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:58:46.992658 1946 raft_consensus.cc:2229] T 34dee7f4687243aaa359bda73a875b04 P c2d58cba7d4847c4901dcd119b6bd6bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:58:46.994925 1946 raft_consensus.cc:2258] T 34dee7f4687243aaa359bda73a875b04 P c2d58cba7d4847c4901dcd119b6bd6bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:58:46.995080 1516 tablet_service.cc:1467] Processing CreateTablet for tablet 04071b8705d4451e835162a19c8ced60 (DEFAULT_TABLE table=test-table [id=e3d5f1cbcba94bb980cb5eb0c5e3cf2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:58:46.995699 1516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04071b8705d4451e835162a19c8ced60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:58:46.996495 1802 tablet_service.cc:1467] Processing CreateTablet for tablet 04071b8705d4451e835162a19c8ced60 (DEFAULT_TABLE table=test-table [id=e3d5f1cbcba94bb980cb5eb0c5e3cf2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:58:46.996863 1659 tablet_service.cc:1467] Processing CreateTablet for tablet 04071b8705d4451e835162a19c8ced60 (DEFAULT_TABLE table=test-table [id=e3d5f1cbcba94bb980cb5eb0c5e3cf2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:58:46.997444 1659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04071b8705d4451e835162a19c8ced60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:58:46.999020 1802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04071b8705d4451e835162a19c8ced60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:58:47.003652 1893 tablet_bootstrap.cc:492] T 04071b8705d4451e835162a19c8ced60 P 451b2658f7d946c8a4cc453e96d8613f: Bootstrap starting.
I20260806 02:58:47.005182 1947 ts_tablet_manager.cc:1905] T 34dee7f4687243aaa359bda73a875b04 P f0ac4bacaea845ad987b992b68df2772: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:58:47.007279 1893 tablet_bootstrap.cc:654] T 04071b8705d4451e835162a19c8ced60 P 451b2658f7d946c8a4cc453e96d8613f: Neither blocks nor log segments found. Creating new log.
I20260806 02:58:47.008351 1945 tablet_replica.cc:330] T 34dee7f4687243aaa359bda73a875b04 P 451b2658f7d946c8a4cc453e96d8613f: stopping tablet replica
I20260806 02:58:47.008958 1945 raft_consensus.cc:2229] T 34dee7f4687243aaa359bda73a875b04 P 451b2658f7d946c8a4cc453e96d8613f [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:58:47.011382 1945 raft_consensus.cc:2258] T 34dee7f4687243aaa359bda73a875b04 P 451b2658f7d946c8a4cc453e96d8613f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:58:47.015996 1945 ts_tablet_manager.cc:1905] T 34dee7f4687243aaa359bda73a875b04 P 451b2658f7d946c8a4cc453e96d8613f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:58:47.018225 1946 ts_tablet_manager.cc:1905] T 34dee7f4687243aaa359bda73a875b04 P c2d58cba7d4847c4901dcd119b6bd6bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:58:47.018517 1892 tablet_bootstrap.cc:492] T 04071b8705d4451e835162a19c8ced60 P f0ac4bacaea845ad987b992b68df2772: Bootstrap starting.
I20260806 02:58:47.020562 1892 tablet_bootstrap.cc:654] T 04071b8705d4451e835162a19c8ced60 P f0ac4bacaea845ad987b992b68df2772: Neither blocks nor log segments found. Creating new log.
I20260806 02:58:47.023175 1947 ts_tablet_manager.cc:1918] T 34dee7f4687243aaa359bda73a875b04 P f0ac4bacaea845ad987b992b68df2772: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:58:47.023447 1947 log.cc:1196] T 34dee7f4687243aaa359bda73a875b04 P f0ac4bacaea845ad987b992b68df2772: Deleting WAL directory at /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/minicluster-data/ts-2/wal/wals/34dee7f4687243aaa359bda73a875b04
I20260806 02:58:47.024487 1947 ts_tablet_manager.cc:1939] T 34dee7f4687243aaa359bda73a875b04 P f0ac4bacaea845ad987b992b68df2772: Deleting consensus metadata
I20260806 02:58:47.027032 1364 catalog_manager.cc:4841] TS f0ac4bacaea845ad987b992b68df2772 (127.31.74.195:39003): tablet 34dee7f4687243aaa359bda73a875b04 (table test-table [id=48cb84502c134ca2998fd8edb9098420]) successfully deleted
I20260806 02:58:47.028460 1893 tablet_bootstrap.cc:492] T 04071b8705d4451e835162a19c8ced60 P 451b2658f7d946c8a4cc453e96d8613f: No bootstrap required, opened a new log
I20260806 02:58:47.028716 1893 ts_tablet_manager.cc:1397] T 04071b8705d4451e835162a19c8ced60 P 451b2658f7d946c8a4cc453e96d8613f: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.004s
I20260806 02:58:47.029008 1892 tablet_bootstrap.cc:492] T 04071b8705d4451e835162a19c8ced60 P f0ac4bacaea845ad987b992b68df2772: No bootstrap required, opened a new log
I20260806 02:58:47.029214 1892 ts_tablet_manager.cc:1397] T 04071b8705d4451e835162a19c8ced60 P f0ac4bacaea845ad987b992b68df2772: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.003s
I20260806 02:58:47.029654 1893 raft_consensus.cc:348] T 04071b8705d4451e835162a19c8ced60 P 451b2658f7d946c8a4cc453e96d8613f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451b2658f7d946c8a4cc453e96d8613f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37965 } } peers { permanent_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 36655 } } peers { permanent_uuid: "f0ac4bacaea845ad987b992b68df2772" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 39003 } }
I20260806 02:58:47.029950 1893 raft_consensus.cc:374] T 04071b8705d4451e835162a19c8ced60 P 451b2658f7d946c8a4cc453e96d8613f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:58:47.030053 1893 raft_consensus.cc:729] T 04071b8705d4451e835162a19c8ced60 P 451b2658f7d946c8a4cc453e96d8613f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 451b2658f7d946c8a4cc453e96d8613f, State: Initialized, Role: FOLLOWER
I20260806 02:58:47.030227 1892 raft_consensus.cc:348] T 04071b8705d4451e835162a19c8ced60 P f0ac4bacaea845ad987b992b68df2772 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451b2658f7d946c8a4cc453e96d8613f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37965 } } peers { permanent_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 36655 } } peers { permanent_uuid: "f0ac4bacaea845ad987b992b68df2772" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 39003 } }
I20260806 02:58:47.030819 1892 raft_consensus.cc:374] T 04071b8705d4451e835162a19c8ced60 P f0ac4bacaea845ad987b992b68df2772 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:58:47.030915 1892 raft_consensus.cc:729] T 04071b8705d4451e835162a19c8ced60 P f0ac4bacaea845ad987b992b68df2772 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0ac4bacaea845ad987b992b68df2772, State: Initialized, Role: FOLLOWER
I20260806 02:58:47.030706 1893 consensus_queue.cc:260] T 04071b8705d4451e835162a19c8ced60 P 451b2658f7d946c8a4cc453e96d8613f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451b2658f7d946c8a4cc453e96d8613f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37965 } } peers { permanent_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 36655 } } peers { permanent_uuid: "f0ac4bacaea845ad987b992b68df2772" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 39003 } }
I20260806 02:58:47.031196 1892 consensus_queue.cc:260] T 04071b8705d4451e835162a19c8ced60 P f0ac4bacaea845ad987b992b68df2772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451b2658f7d946c8a4cc453e96d8613f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37965 } } peers { permanent_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 36655 } } peers { permanent_uuid: "f0ac4bacaea845ad987b992b68df2772" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 39003 } }
I20260806 02:58:47.031827 1892 ts_tablet_manager.cc:1428] T 04071b8705d4451e835162a19c8ced60 P f0ac4bacaea845ad987b992b68df2772: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260806 02:58:47.038488 1893 ts_tablet_manager.cc:1428] T 04071b8705d4451e835162a19c8ced60 P 451b2658f7d946c8a4cc453e96d8613f: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20260806 02:58:47.041927 1894 tablet_bootstrap.cc:492] T 04071b8705d4451e835162a19c8ced60 P c2d58cba7d4847c4901dcd119b6bd6bb: Bootstrap starting.
I20260806 02:58:47.043605 1945 ts_tablet_manager.cc:1918] T 34dee7f4687243aaa359bda73a875b04 P 451b2658f7d946c8a4cc453e96d8613f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:58:47.043828 1945 log.cc:1196] T 34dee7f4687243aaa359bda73a875b04 P 451b2658f7d946c8a4cc453e96d8613f: Deleting WAL directory at /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/minicluster-data/ts-0/wal/wals/34dee7f4687243aaa359bda73a875b04
I20260806 02:58:47.044061 1894 tablet_bootstrap.cc:654] T 04071b8705d4451e835162a19c8ced60 P c2d58cba7d4847c4901dcd119b6bd6bb: Neither blocks nor log segments found. Creating new log.
I20260806 02:58:47.044462 1945 ts_tablet_manager.cc:1939] T 34dee7f4687243aaa359bda73a875b04 P 451b2658f7d946c8a4cc453e96d8613f: Deleting consensus metadata
I20260806 02:58:47.046722 1946 ts_tablet_manager.cc:1918] T 34dee7f4687243aaa359bda73a875b04 P c2d58cba7d4847c4901dcd119b6bd6bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:58:47.046963 1946 log.cc:1196] T 34dee7f4687243aaa359bda73a875b04 P c2d58cba7d4847c4901dcd119b6bd6bb: Deleting WAL directory at /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/minicluster-data/ts-1/wal/wals/34dee7f4687243aaa359bda73a875b04
I20260806 02:58:47.047866 1946 ts_tablet_manager.cc:1939] T 34dee7f4687243aaa359bda73a875b04 P c2d58cba7d4847c4901dcd119b6bd6bb: Deleting consensus metadata
I20260806 02:58:47.051015 1365 catalog_manager.cc:4841] TS 451b2658f7d946c8a4cc453e96d8613f (127.31.74.193:37965): tablet 34dee7f4687243aaa359bda73a875b04 (table test-table [id=48cb84502c134ca2998fd8edb9098420]) successfully deleted
I20260806 02:58:47.052277 1363 catalog_manager.cc:4841] TS c2d58cba7d4847c4901dcd119b6bd6bb (127.31.74.194:36655): tablet 34dee7f4687243aaa359bda73a875b04 (table test-table [id=48cb84502c134ca2998fd8edb9098420]) successfully deleted
I20260806 02:58:47.052872 1894 tablet_bootstrap.cc:492] T 04071b8705d4451e835162a19c8ced60 P c2d58cba7d4847c4901dcd119b6bd6bb: No bootstrap required, opened a new log
I20260806 02:58:47.053138 1894 ts_tablet_manager.cc:1397] T 04071b8705d4451e835162a19c8ced60 P c2d58cba7d4847c4901dcd119b6bd6bb: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.003s
I20260806 02:58:47.054075 1894 raft_consensus.cc:348] T 04071b8705d4451e835162a19c8ced60 P c2d58cba7d4847c4901dcd119b6bd6bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451b2658f7d946c8a4cc453e96d8613f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37965 } } peers { permanent_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 36655 } } peers { permanent_uuid: "f0ac4bacaea845ad987b992b68df2772" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 39003 } }
I20260806 02:58:47.054431 1894 raft_consensus.cc:374] T 04071b8705d4451e835162a19c8ced60 P c2d58cba7d4847c4901dcd119b6bd6bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:58:47.054550 1894 raft_consensus.cc:729] T 04071b8705d4451e835162a19c8ced60 P c2d58cba7d4847c4901dcd119b6bd6bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2d58cba7d4847c4901dcd119b6bd6bb, State: Initialized, Role: FOLLOWER
I20260806 02:58:47.054955 1894 consensus_queue.cc:260] T 04071b8705d4451e835162a19c8ced60 P c2d58cba7d4847c4901dcd119b6bd6bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451b2658f7d946c8a4cc453e96d8613f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37965 } } peers { permanent_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 36655 } } peers { permanent_uuid: "f0ac4bacaea845ad987b992b68df2772" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 39003 } }
I20260806 02:58:47.056504 1894 ts_tablet_manager.cc:1428] T 04071b8705d4451e835162a19c8ced60 P c2d58cba7d4847c4901dcd119b6bd6bb: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260806 02:58:47.301143 1899 raft_consensus.cc:482] T 04071b8705d4451e835162a19c8ced60 P f0ac4bacaea845ad987b992b68df2772 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:58:47.301689 1899 raft_consensus.cc:504] T 04071b8705d4451e835162a19c8ced60 P f0ac4bacaea845ad987b992b68df2772 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451b2658f7d946c8a4cc453e96d8613f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37965 } } peers { permanent_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 36655 } } peers { permanent_uuid: "f0ac4bacaea845ad987b992b68df2772" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 39003 } }
I20260806 02:58:47.303102 1900 raft_consensus.cc:482] T 04071b8705d4451e835162a19c8ced60 P c2d58cba7d4847c4901dcd119b6bd6bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:58:47.303333 1900 raft_consensus.cc:504] T 04071b8705d4451e835162a19c8ced60 P c2d58cba7d4847c4901dcd119b6bd6bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451b2658f7d946c8a4cc453e96d8613f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37965 } } peers { permanent_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 36655 } } peers { permanent_uuid: "f0ac4bacaea845ad987b992b68df2772" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 39003 } }
I20260806 02:58:47.303486 1537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04071b8705d4451e835162a19c8ced60" candidate_uuid: "f0ac4bacaea845ad987b992b68df2772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "451b2658f7d946c8a4cc453e96d8613f" is_pre_election: true
I20260806 02:58:47.303785 1537 raft_consensus.cc:2454] T 04071b8705d4451e835162a19c8ced60 P 451b2658f7d946c8a4cc453e96d8613f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0ac4bacaea845ad987b992b68df2772 in term 0.
I20260806 02:58:47.304100 1900 leader_election.cc:288] T 04071b8705d4451e835162a19c8ced60 P c2d58cba7d4847c4901dcd119b6bd6bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 451b2658f7d946c8a4cc453e96d8613f (127.31.74.193:37965), f0ac4bacaea845ad987b992b68df2772 (127.31.74.195:39003)
I20260806 02:58:47.304505 1755 leader_election.cc:302] T 04071b8705d4451e835162a19c8ced60 P f0ac4bacaea845ad987b992b68df2772 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 451b2658f7d946c8a4cc453e96d8613f, f0ac4bacaea845ad987b992b68df2772; no voters:
I20260806 02:58:47.304986 1899 leader_election.cc:288] T 04071b8705d4451e835162a19c8ced60 P f0ac4bacaea845ad987b992b68df2772 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 451b2658f7d946c8a4cc453e96d8613f (127.31.74.193:37965), c2d58cba7d4847c4901dcd119b6bd6bb (127.31.74.194:36655)
I20260806 02:58:47.305156 1899 raft_consensus.cc:2781] T 04071b8705d4451e835162a19c8ced60 P f0ac4bacaea845ad987b992b68df2772 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:58:47.305294 1899 raft_consensus.cc:482] T 04071b8705d4451e835162a19c8ced60 P f0ac4bacaea845ad987b992b68df2772 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:58:47.305382 1899 raft_consensus.cc:3037] T 04071b8705d4451e835162a19c8ced60 P f0ac4bacaea845ad987b992b68df2772 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:58:47.305948 1537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04071b8705d4451e835162a19c8ced60" candidate_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "451b2658f7d946c8a4cc453e96d8613f" is_pre_election: true
I20260806 02:58:47.306244 1537 raft_consensus.cc:2454] T 04071b8705d4451e835162a19c8ced60 P 451b2658f7d946c8a4cc453e96d8613f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2d58cba7d4847c4901dcd119b6bd6bb in term 0.
I20260806 02:58:47.306806 1611 leader_election.cc:302] T 04071b8705d4451e835162a19c8ced60 P c2d58cba7d4847c4901dcd119b6bd6bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 451b2658f7d946c8a4cc453e96d8613f, c2d58cba7d4847c4901dcd119b6bd6bb; no voters:
I20260806 02:58:47.307708 1900 raft_consensus.cc:2781] T 04071b8705d4451e835162a19c8ced60 P c2d58cba7d4847c4901dcd119b6bd6bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:58:47.307853 1900 raft_consensus.cc:482] T 04071b8705d4451e835162a19c8ced60 P c2d58cba7d4847c4901dcd119b6bd6bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:58:47.307977 1900 raft_consensus.cc:3037] T 04071b8705d4451e835162a19c8ced60 P c2d58cba7d4847c4901dcd119b6bd6bb [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:58:47.309238 1899 raft_consensus.cc:504] T 04071b8705d4451e835162a19c8ced60 P f0ac4bacaea845ad987b992b68df2772 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451b2658f7d946c8a4cc453e96d8613f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37965 } } peers { permanent_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 36655 } } peers { permanent_uuid: "f0ac4bacaea845ad987b992b68df2772" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 39003 } }
I20260806 02:58:47.309988 1899 leader_election.cc:288] T 04071b8705d4451e835162a19c8ced60 P f0ac4bacaea845ad987b992b68df2772 [CANDIDATE]: Term 1 election: Requested vote from peers 451b2658f7d946c8a4cc453e96d8613f (127.31.74.193:37965), c2d58cba7d4847c4901dcd119b6bd6bb (127.31.74.194:36655)
I20260806 02:58:47.310204 1822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04071b8705d4451e835162a19c8ced60" candidate_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0ac4bacaea845ad987b992b68df2772" is_pre_election: true
I20260806 02:58:47.310509 1822 raft_consensus.cc:2379] T 04071b8705d4451e835162a19c8ced60 P f0ac4bacaea845ad987b992b68df2772 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c2d58cba7d4847c4901dcd119b6bd6bb in current term 1: Already voted for candidate f0ac4bacaea845ad987b992b68df2772 in this term.
I20260806 02:58:47.312275 1679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04071b8705d4451e835162a19c8ced60" candidate_uuid: "f0ac4bacaea845ad987b992b68df2772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb"
I20260806 02:58:47.312242 1680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04071b8705d4451e835162a19c8ced60" candidate_uuid: "f0ac4bacaea845ad987b992b68df2772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" is_pre_election: true
I20260806 02:58:47.314971 1900 raft_consensus.cc:504] T 04071b8705d4451e835162a19c8ced60 P c2d58cba7d4847c4901dcd119b6bd6bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451b2658f7d946c8a4cc453e96d8613f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37965 } } peers { permanent_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 36655 } } peers { permanent_uuid: "f0ac4bacaea845ad987b992b68df2772" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 39003 } }
I20260806 02:58:47.316217 1679 raft_consensus.cc:2379] T 04071b8705d4451e835162a19c8ced60 P c2d58cba7d4847c4901dcd119b6bd6bb [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f0ac4bacaea845ad987b992b68df2772 in current term 1: Already voted for candidate c2d58cba7d4847c4901dcd119b6bd6bb in this term.
I20260806 02:58:47.318714 1537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04071b8705d4451e835162a19c8ced60" candidate_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "451b2658f7d946c8a4cc453e96d8613f"
I20260806 02:58:47.319015 1537 raft_consensus.cc:3037] T 04071b8705d4451e835162a19c8ced60 P 451b2658f7d946c8a4cc453e96d8613f [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:58:47.323076 1537 raft_consensus.cc:2454] T 04071b8705d4451e835162a19c8ced60 P 451b2658f7d946c8a4cc453e96d8613f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2d58cba7d4847c4901dcd119b6bd6bb in term 1.
I20260806 02:58:47.323876 1611 leader_election.cc:302] T 04071b8705d4451e835162a19c8ced60 P c2d58cba7d4847c4901dcd119b6bd6bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 451b2658f7d946c8a4cc453e96d8613f, c2d58cba7d4847c4901dcd119b6bd6bb; no voters:
I20260806 02:58:47.324581 1917 raft_consensus.cc:2781] T 04071b8705d4451e835162a19c8ced60 P c2d58cba7d4847c4901dcd119b6bd6bb [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:58:47.324777 1917 raft_consensus.cc:686] T 04071b8705d4451e835162a19c8ced60 P c2d58cba7d4847c4901dcd119b6bd6bb [term 1 LEADER]: Becoming Leader. State: Replica: c2d58cba7d4847c4901dcd119b6bd6bb, State: Running, Role: LEADER
I20260806 02:58:47.325114 1917 consensus_queue.cc:237] T 04071b8705d4451e835162a19c8ced60 P c2d58cba7d4847c4901dcd119b6bd6bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451b2658f7d946c8a4cc453e96d8613f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37965 } } peers { permanent_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 36655 } } peers { permanent_uuid: "f0ac4bacaea845ad987b992b68df2772" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 39003 } }
I20260806 02:58:47.329442 1900 leader_election.cc:288] T 04071b8705d4451e835162a19c8ced60 P c2d58cba7d4847c4901dcd119b6bd6bb [CANDIDATE]: Term 1 election: Requested vote from peers 451b2658f7d946c8a4cc453e96d8613f (127.31.74.193:37965), f0ac4bacaea845ad987b992b68df2772 (127.31.74.195:39003)
I20260806 02:58:47.330070 1822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04071b8705d4451e835162a19c8ced60" candidate_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0ac4bacaea845ad987b992b68df2772"
I20260806 02:58:47.330452 1822 raft_consensus.cc:2379] T 04071b8705d4451e835162a19c8ced60 P f0ac4bacaea845ad987b992b68df2772 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate c2d58cba7d4847c4901dcd119b6bd6bb in current term 1: Already voted for candidate f0ac4bacaea845ad987b992b68df2772 in this term.
I20260806 02:58:47.335896 1537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04071b8705d4451e835162a19c8ced60" candidate_uuid: "f0ac4bacaea845ad987b992b68df2772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "451b2658f7d946c8a4cc453e96d8613f"
I20260806 02:58:47.336354 1537 raft_consensus.cc:2379] T 04071b8705d4451e835162a19c8ced60 P 451b2658f7d946c8a4cc453e96d8613f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f0ac4bacaea845ad987b992b68df2772 in current term 1: Already voted for candidate c2d58cba7d4847c4901dcd119b6bd6bb in this term.
I20260806 02:58:47.336973 1755 leader_election.cc:302] T 04071b8705d4451e835162a19c8ced60 P f0ac4bacaea845ad987b992b68df2772 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0ac4bacaea845ad987b992b68df2772; no voters: 451b2658f7d946c8a4cc453e96d8613f, c2d58cba7d4847c4901dcd119b6bd6bb
I20260806 02:58:47.337496 1951 raft_consensus.cc:2726] T 04071b8705d4451e835162a19c8ced60 P f0ac4bacaea845ad987b992b68df2772 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260806 02:58:47.337903 1378 catalog_manager.cc:5495] T 04071b8705d4451e835162a19c8ced60 P c2d58cba7d4847c4901dcd119b6bd6bb reported cstate change: term changed from 0 to 1, leader changed from <none> to c2d58cba7d4847c4901dcd119b6bd6bb (127.31.74.194). New cstate: current_term: 1 leader_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451b2658f7d946c8a4cc453e96d8613f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 36655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0ac4bacaea845ad987b992b68df2772" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 39003 } health_report { overall_health: UNKNOWN } } }
I20260806 02:58:47.408491 1955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:47.400919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55704 (local address 127.31.74.194:36655)
0806 02:58:47.401685 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:47.401694 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:47.401719 (+ 25us) server_negotiation.cc:407] Connection header received
0806 02:58:47.401797 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:47.401803 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:47.401887 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:47.402004 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:58:47.402433 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:47.403394 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:47.405289 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:47.406015 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:47.406168 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:47.406965 (+ 797us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:58:47.407127 (+ 162us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:58:47.407230 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:47.407855 (+ 625us) server_negotiation.cc:299] Negotiation successful
0806 02:58:47.407958 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":518,"threads_started":1}
I20260806 02:58:47.427594 1955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:47.420261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.194:45153 (local address 127.31.74.194:36655)
0806 02:58:47.420596 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:47.420603 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:47.420901 (+ 298us) server_negotiation.cc:407] Connection header received
0806 02:58:47.421117 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:47.421122 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:47.421201 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:47.421320 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:58:47.421998 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:47.423280 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:47.425045 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:47.425796 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:47.425950 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:47.426504 (+ 554us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:58:47.426707 (+ 203us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:58:47.426849 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:47.427017 (+ 168us) server_negotiation.cc:299] Negotiation successful
0806 02:58:47.427155 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":89}
I20260806 02:58:47.427721 1958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:47.420143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.194:36655 (local address 127.31.74.194:45153)
0806 02:58:47.420796 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:47.420812 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:58:47.420980 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:58:47.421332 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:58:47.421337 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:58:47.421348 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 02:58:47.421696 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:47.421715 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:47.423471 (+ 1756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:47.423476 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:47.424884 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:47.424898 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:47.426027 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:47.426032 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:47.426243 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:47.426298 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 02:58:47.426859 (+ 561us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 02:58:47.426871 (+ 12us) client_negotiation.cc:769] Sending connection context
0806 02:58:47.426962 (+ 91us) client_negotiation.cc:240] Negotiation successful
0806 02:58:47.427120 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"mutex_wait_us":3,"thread_start_us":334,"threads_started":1}
I20260806 02:58:47.441635 1822 raft_consensus.cc:1261] T 04071b8705d4451e835162a19c8ced60 P f0ac4bacaea845ad987b992b68df2772 [term 1 FOLLOWER]: Refusing update from remote peer c2d58cba7d4847c4901dcd119b6bd6bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:58:47.441983 1537 raft_consensus.cc:1261] T 04071b8705d4451e835162a19c8ced60 P 451b2658f7d946c8a4cc453e96d8613f [term 1 FOLLOWER]: Refusing update from remote peer c2d58cba7d4847c4901dcd119b6bd6bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:58:47.442565 1943 consensus_queue.cc:1035] T 04071b8705d4451e835162a19c8ced60 P c2d58cba7d4847c4901dcd119b6bd6bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0ac4bacaea845ad987b992b68df2772" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 39003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:58:47.443116 1917 consensus_queue.cc:1035] T 04071b8705d4451e835162a19c8ced60 P c2d58cba7d4847c4901dcd119b6bd6bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "451b2658f7d946c8a4cc453e96d8613f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 02:58:47.480029 1922 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260806 02:58:49.118966 1378 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36838:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:58:49.119310 1378 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36838:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:58:49.124805 1378 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 24ba9714cb1a494d983288e0ab1b4283: Sending DeleteTablet for 3 replicas of tablet 04071b8705d4451e835162a19c8ced60
I20260806 02:58:49.125981 1517 tablet_service.cc:1514] Processing DeleteTablet for tablet 04071b8705d4451e835162a19c8ced60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:58:49 UTC) from {username='kudu', principal='kudu/127.31.74.254@KRBTEST.COM'} at 127.0.0.1:36948
I20260806 02:58:49.126153 1658 tablet_service.cc:1514] Processing DeleteTablet for tablet 04071b8705d4451e835162a19c8ced60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:58:49 UTC) from {username='kudu', principal='kudu/127.31.74.254@KRBTEST.COM'} at 127.0.0.1:37248
I20260806 02:58:49.126151 1802 tablet_service.cc:1514] Processing DeleteTablet for tablet 04071b8705d4451e835162a19c8ced60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:58:49 UTC) from {username='kudu', principal='kudu/127.31.74.254@KRBTEST.COM'} at 127.0.0.1:60902
I20260806 02:58:49.137409 1985 tablet_replica.cc:330] T 04071b8705d4451e835162a19c8ced60 P f0ac4bacaea845ad987b992b68df2772: stopping tablet replica
I20260806 02:58:49.137409 1986 tablet_replica.cc:330] T 04071b8705d4451e835162a19c8ced60 P c2d58cba7d4847c4901dcd119b6bd6bb: stopping tablet replica
I20260806 02:58:49.137724 1986 raft_consensus.cc:2229] T 04071b8705d4451e835162a19c8ced60 P c2d58cba7d4847c4901dcd119b6bd6bb [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:58:49.137876 1985 raft_consensus.cc:2229] T 04071b8705d4451e835162a19c8ced60 P f0ac4bacaea845ad987b992b68df2772 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:58:49.138264 1985 raft_consensus.cc:2258] T 04071b8705d4451e835162a19c8ced60 P f0ac4bacaea845ad987b992b68df2772 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:58:49.138561 1984 tablet_replica.cc:330] T 04071b8705d4451e835162a19c8ced60 P 451b2658f7d946c8a4cc453e96d8613f: stopping tablet replica
I20260806 02:58:49.138617 1986 raft_consensus.cc:2258] T 04071b8705d4451e835162a19c8ced60 P c2d58cba7d4847c4901dcd119b6bd6bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:58:49.138983 1984 raft_consensus.cc:2229] T 04071b8705d4451e835162a19c8ced60 P 451b2658f7d946c8a4cc453e96d8613f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:58:49.139374 1984 raft_consensus.cc:2258] T 04071b8705d4451e835162a19c8ced60 P 451b2658f7d946c8a4cc453e96d8613f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:58:49.139984 1985 ts_tablet_manager.cc:1905] T 04071b8705d4451e835162a19c8ced60 P f0ac4bacaea845ad987b992b68df2772: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:58:49.140134 1986 ts_tablet_manager.cc:1905] T 04071b8705d4451e835162a19c8ced60 P c2d58cba7d4847c4901dcd119b6bd6bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:58:49.140450 1984 ts_tablet_manager.cc:1905] T 04071b8705d4451e835162a19c8ced60 P 451b2658f7d946c8a4cc453e96d8613f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:58:49.146245 1986 ts_tablet_manager.cc:1918] T 04071b8705d4451e835162a19c8ced60 P c2d58cba7d4847c4901dcd119b6bd6bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 02:58:49.146447 1986 log.cc:1196] T 04071b8705d4451e835162a19c8ced60 P c2d58cba7d4847c4901dcd119b6bd6bb: Deleting WAL directory at /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/minicluster-data/ts-1/wal/wals/04071b8705d4451e835162a19c8ced60
I20260806 02:58:49.147202 1986 ts_tablet_manager.cc:1939] T 04071b8705d4451e835162a19c8ced60 P c2d58cba7d4847c4901dcd119b6bd6bb: Deleting consensus metadata
I20260806 02:58:49.148764 1363 catalog_manager.cc:4841] TS c2d58cba7d4847c4901dcd119b6bd6bb (127.31.74.194:36655): tablet 04071b8705d4451e835162a19c8ced60 (table test-table [id=e3d5f1cbcba94bb980cb5eb0c5e3cf2a]) successfully deleted
I20260806 02:58:49.148919 1985 ts_tablet_manager.cc:1918] T 04071b8705d4451e835162a19c8ced60 P f0ac4bacaea845ad987b992b68df2772: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 02:58:49.149032 1984 ts_tablet_manager.cc:1918] T 04071b8705d4451e835162a19c8ced60 P 451b2658f7d946c8a4cc453e96d8613f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 02:58:49.149067 1985 log.cc:1196] T 04071b8705d4451e835162a19c8ced60 P f0ac4bacaea845ad987b992b68df2772: Deleting WAL directory at /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/minicluster-data/ts-2/wal/wals/04071b8705d4451e835162a19c8ced60
I20260806 02:58:49.149190 1984 log.cc:1196] T 04071b8705d4451e835162a19c8ced60 P 451b2658f7d946c8a4cc453e96d8613f: Deleting WAL directory at /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/minicluster-data/ts-0/wal/wals/04071b8705d4451e835162a19c8ced60
I20260806 02:58:49.149591 1985 ts_tablet_manager.cc:1939] T 04071b8705d4451e835162a19c8ced60 P f0ac4bacaea845ad987b992b68df2772: Deleting consensus metadata
I20260806 02:58:49.149657 1984 ts_tablet_manager.cc:1939] T 04071b8705d4451e835162a19c8ced60 P 451b2658f7d946c8a4cc453e96d8613f: Deleting consensus metadata
I20260806 02:58:49.151044 1364 catalog_manager.cc:4841] TS f0ac4bacaea845ad987b992b68df2772 (127.31.74.195:39003): tablet 04071b8705d4451e835162a19c8ced60 (table test-table [id=e3d5f1cbcba94bb980cb5eb0c5e3cf2a]) successfully deleted
I20260806 02:58:49.151086 1365 catalog_manager.cc:4841] TS 451b2658f7d946c8a4cc453e96d8613f (127.31.74.193:37965): tablet 04071b8705d4451e835162a19c8ced60 (table test-table [id=e3d5f1cbcba94bb980cb5eb0c5e3cf2a]) successfully deleted
Aug 06 02:58:49 dist-test-slave-dhph krb5kdc[1333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985129, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 02:58:49 dist-test-slave-dhph krb5kdc[1333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985129, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 02:58:49.213938 1994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:49.187309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51946 (local address 127.31.74.254:42053)
0806 02:58:49.188204 (+ 895us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:49.188209 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:49.188235 (+ 26us) server_negotiation.cc:407] Connection header received
0806 02:58:49.188292 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:49.188295 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:49.188358 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:49.188455 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:49.190371 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:49.191392 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:49.193220 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:49.193725 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:49.193856 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:49.205909 (+ 12053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:58:49.205916 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:58:49.205919 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:58:49.205967 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:58:49.210216 (+ 4249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:49.211342 (+ 1126us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:49.211348 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:49.211350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:49.211501 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:49.212158 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:49.212163 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:49.212165 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:49.212493 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:58:49.212727 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:49.213200 (+ 473us) server_negotiation.cc:299] Negotiation successful
0806 02:58:49.213412 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":783,"thread_start_us":630,"threads_started":1}
I20260806 02:58:49.219208 1378 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51946:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:58:49.219905 1378 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:58:49.228891 1658 tablet_service.cc:1467] Processing CreateTablet for tablet e629f8104e324abbbcea3b2a8022f973 (DEFAULT_TABLE table=test-table [id=4e11796a27454b1f92e2b04b28290f4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:58:49.228915 1517 tablet_service.cc:1467] Processing CreateTablet for tablet e629f8104e324abbbcea3b2a8022f973 (DEFAULT_TABLE table=test-table [id=4e11796a27454b1f92e2b04b28290f4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:58:49.228999 1802 tablet_service.cc:1467] Processing CreateTablet for tablet e629f8104e324abbbcea3b2a8022f973 (DEFAULT_TABLE table=test-table [id=4e11796a27454b1f92e2b04b28290f4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:58:49.229413 1517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e629f8104e324abbbcea3b2a8022f973. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:58:49.229452 1658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e629f8104e324abbbcea3b2a8022f973. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:58:49.229501 1802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e629f8104e324abbbcea3b2a8022f973. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:58:49.241580 1997 tablet_bootstrap.cc:492] T e629f8104e324abbbcea3b2a8022f973 P f0ac4bacaea845ad987b992b68df2772: Bootstrap starting.
I20260806 02:58:49.244586 1997 tablet_bootstrap.cc:654] T e629f8104e324abbbcea3b2a8022f973 P f0ac4bacaea845ad987b992b68df2772: Neither blocks nor log segments found. Creating new log.
I20260806 02:58:49.247054 1995 tablet_bootstrap.cc:492] T e629f8104e324abbbcea3b2a8022f973 P c2d58cba7d4847c4901dcd119b6bd6bb: Bootstrap starting.
I20260806 02:58:49.248953 1996 tablet_bootstrap.cc:492] T e629f8104e324abbbcea3b2a8022f973 P 451b2658f7d946c8a4cc453e96d8613f: Bootstrap starting.
I20260806 02:58:49.249612 1997 tablet_bootstrap.cc:492] T e629f8104e324abbbcea3b2a8022f973 P f0ac4bacaea845ad987b992b68df2772: No bootstrap required, opened a new log
I20260806 02:58:49.249918 1997 ts_tablet_manager.cc:1397] T e629f8104e324abbbcea3b2a8022f973 P f0ac4bacaea845ad987b992b68df2772: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260806 02:58:49.250953 1997 raft_consensus.cc:348] T e629f8104e324abbbcea3b2a8022f973 P f0ac4bacaea845ad987b992b68df2772 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451b2658f7d946c8a4cc453e96d8613f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37965 } } peers { permanent_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 36655 } } peers { permanent_uuid: "f0ac4bacaea845ad987b992b68df2772" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 39003 } }
I20260806 02:58:49.251236 1997 raft_consensus.cc:374] T e629f8104e324abbbcea3b2a8022f973 P f0ac4bacaea845ad987b992b68df2772 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:58:49.251415 1997 raft_consensus.cc:729] T e629f8104e324abbbcea3b2a8022f973 P f0ac4bacaea845ad987b992b68df2772 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0ac4bacaea845ad987b992b68df2772, State: Initialized, Role: FOLLOWER
I20260806 02:58:49.251518 1996 tablet_bootstrap.cc:654] T e629f8104e324abbbcea3b2a8022f973 P 451b2658f7d946c8a4cc453e96d8613f: Neither blocks nor log segments found. Creating new log.
I20260806 02:58:49.251744 1997 consensus_queue.cc:260] T e629f8104e324abbbcea3b2a8022f973 P f0ac4bacaea845ad987b992b68df2772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451b2658f7d946c8a4cc453e96d8613f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37965 } } peers { permanent_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 36655 } } peers { permanent_uuid: "f0ac4bacaea845ad987b992b68df2772" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 39003 } }
I20260806 02:58:49.252964 1997 ts_tablet_manager.cc:1428] T e629f8104e324abbbcea3b2a8022f973 P f0ac4bacaea845ad987b992b68df2772: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260806 02:58:49.254163 1995 tablet_bootstrap.cc:654] T e629f8104e324abbbcea3b2a8022f973 P c2d58cba7d4847c4901dcd119b6bd6bb: Neither blocks nor log segments found. Creating new log.
I20260806 02:58:49.264212 1996 tablet_bootstrap.cc:492] T e629f8104e324abbbcea3b2a8022f973 P 451b2658f7d946c8a4cc453e96d8613f: No bootstrap required, opened a new log
I20260806 02:58:49.264235 1995 tablet_bootstrap.cc:492] T e629f8104e324abbbcea3b2a8022f973 P c2d58cba7d4847c4901dcd119b6bd6bb: No bootstrap required, opened a new log
I20260806 02:58:49.264423 1995 ts_tablet_manager.cc:1397] T e629f8104e324abbbcea3b2a8022f973 P c2d58cba7d4847c4901dcd119b6bd6bb: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.007s
I20260806 02:58:49.264441 1996 ts_tablet_manager.cc:1397] T e629f8104e324abbbcea3b2a8022f973 P 451b2658f7d946c8a4cc453e96d8613f: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.003s
I20260806 02:58:49.265280 1996 raft_consensus.cc:348] T e629f8104e324abbbcea3b2a8022f973 P 451b2658f7d946c8a4cc453e96d8613f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451b2658f7d946c8a4cc453e96d8613f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37965 } } peers { permanent_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 36655 } } peers { permanent_uuid: "f0ac4bacaea845ad987b992b68df2772" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 39003 } }
I20260806 02:58:49.265338 1995 raft_consensus.cc:348] T e629f8104e324abbbcea3b2a8022f973 P c2d58cba7d4847c4901dcd119b6bd6bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451b2658f7d946c8a4cc453e96d8613f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37965 } } peers { permanent_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 36655 } } peers { permanent_uuid: "f0ac4bacaea845ad987b992b68df2772" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 39003 } }
I20260806 02:58:49.265591 1995 raft_consensus.cc:374] T e629f8104e324abbbcea3b2a8022f973 P c2d58cba7d4847c4901dcd119b6bd6bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:58:49.265599 1996 raft_consensus.cc:374] T e629f8104e324abbbcea3b2a8022f973 P 451b2658f7d946c8a4cc453e96d8613f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:58:49.265689 1995 raft_consensus.cc:729] T e629f8104e324abbbcea3b2a8022f973 P c2d58cba7d4847c4901dcd119b6bd6bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2d58cba7d4847c4901dcd119b6bd6bb, State: Initialized, Role: FOLLOWER
I20260806 02:58:49.265707 1996 raft_consensus.cc:729] T e629f8104e324abbbcea3b2a8022f973 P 451b2658f7d946c8a4cc453e96d8613f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 451b2658f7d946c8a4cc453e96d8613f, State: Initialized, Role: FOLLOWER
I20260806 02:58:49.265985 1996 consensus_queue.cc:260] T e629f8104e324abbbcea3b2a8022f973 P 451b2658f7d946c8a4cc453e96d8613f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451b2658f7d946c8a4cc453e96d8613f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37965 } } peers { permanent_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 36655 } } peers { permanent_uuid: "f0ac4bacaea845ad987b992b68df2772" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 39003 } }
I20260806 02:58:49.266057 1995 consensus_queue.cc:260] T e629f8104e324abbbcea3b2a8022f973 P c2d58cba7d4847c4901dcd119b6bd6bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451b2658f7d946c8a4cc453e96d8613f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37965 } } peers { permanent_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 36655 } } peers { permanent_uuid: "f0ac4bacaea845ad987b992b68df2772" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 39003 } }
I20260806 02:58:49.266784 1995 ts_tablet_manager.cc:1428] T e629f8104e324abbbcea3b2a8022f973 P c2d58cba7d4847c4901dcd119b6bd6bb: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260806 02:58:49.267817 1996 ts_tablet_manager.cc:1428] T e629f8104e324abbbcea3b2a8022f973 P 451b2658f7d946c8a4cc453e96d8613f: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260806 02:58:49.541612 1966 raft_consensus.cc:482] T e629f8104e324abbbcea3b2a8022f973 P c2d58cba7d4847c4901dcd119b6bd6bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:58:49.541846 1966 raft_consensus.cc:504] T e629f8104e324abbbcea3b2a8022f973 P c2d58cba7d4847c4901dcd119b6bd6bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451b2658f7d946c8a4cc453e96d8613f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37965 } } peers { permanent_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 36655 } } peers { permanent_uuid: "f0ac4bacaea845ad987b992b68df2772" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 39003 } }
I20260806 02:58:49.542548 1966 leader_election.cc:288] T e629f8104e324abbbcea3b2a8022f973 P c2d58cba7d4847c4901dcd119b6bd6bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 451b2658f7d946c8a4cc453e96d8613f (127.31.74.193:37965), f0ac4bacaea845ad987b992b68df2772 (127.31.74.195:39003)
I20260806 02:58:49.543280 1821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e629f8104e324abbbcea3b2a8022f973" candidate_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0ac4bacaea845ad987b992b68df2772" is_pre_election: true
I20260806 02:58:49.543471 1537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e629f8104e324abbbcea3b2a8022f973" candidate_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "451b2658f7d946c8a4cc453e96d8613f" is_pre_election: true
I20260806 02:58:49.543726 1537 raft_consensus.cc:2454] T e629f8104e324abbbcea3b2a8022f973 P 451b2658f7d946c8a4cc453e96d8613f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2d58cba7d4847c4901dcd119b6bd6bb in term 0.
I20260806 02:58:49.543820 1821 raft_consensus.cc:2454] T e629f8104e324abbbcea3b2a8022f973 P f0ac4bacaea845ad987b992b68df2772 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2d58cba7d4847c4901dcd119b6bd6bb in term 0.
I20260806 02:58:49.544250 1611 leader_election.cc:302] T e629f8104e324abbbcea3b2a8022f973 P c2d58cba7d4847c4901dcd119b6bd6bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 451b2658f7d946c8a4cc453e96d8613f, c2d58cba7d4847c4901dcd119b6bd6bb; no voters:
I20260806 02:58:49.544660 1966 raft_consensus.cc:2781] T e629f8104e324abbbcea3b2a8022f973 P c2d58cba7d4847c4901dcd119b6bd6bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:58:49.544822 1966 raft_consensus.cc:482] T e629f8104e324abbbcea3b2a8022f973 P c2d58cba7d4847c4901dcd119b6bd6bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:58:49.544999 1966 raft_consensus.cc:3037] T e629f8104e324abbbcea3b2a8022f973 P c2d58cba7d4847c4901dcd119b6bd6bb [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:58:49.547252 1966 raft_consensus.cc:504] T e629f8104e324abbbcea3b2a8022f973 P c2d58cba7d4847c4901dcd119b6bd6bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451b2658f7d946c8a4cc453e96d8613f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37965 } } peers { permanent_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 36655 } } peers { permanent_uuid: "f0ac4bacaea845ad987b992b68df2772" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 39003 } }
I20260806 02:58:49.547888 1966 leader_election.cc:288] T e629f8104e324abbbcea3b2a8022f973 P c2d58cba7d4847c4901dcd119b6bd6bb [CANDIDATE]: Term 1 election: Requested vote from peers 451b2658f7d946c8a4cc453e96d8613f (127.31.74.193:37965), f0ac4bacaea845ad987b992b68df2772 (127.31.74.195:39003)
I20260806 02:58:49.548863 1537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e629f8104e324abbbcea3b2a8022f973" candidate_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "451b2658f7d946c8a4cc453e96d8613f"
I20260806 02:58:49.549322 1537 raft_consensus.cc:3037] T e629f8104e324abbbcea3b2a8022f973 P 451b2658f7d946c8a4cc453e96d8613f [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:58:49.549096 1821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e629f8104e324abbbcea3b2a8022f973" candidate_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0ac4bacaea845ad987b992b68df2772"
I20260806 02:58:49.549676 1821 raft_consensus.cc:3037] T e629f8104e324abbbcea3b2a8022f973 P f0ac4bacaea845ad987b992b68df2772 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:58:49.552183 1537 raft_consensus.cc:2454] T e629f8104e324abbbcea3b2a8022f973 P 451b2658f7d946c8a4cc453e96d8613f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2d58cba7d4847c4901dcd119b6bd6bb in term 1.
I20260806 02:58:49.552600 1821 raft_consensus.cc:2454] T e629f8104e324abbbcea3b2a8022f973 P f0ac4bacaea845ad987b992b68df2772 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2d58cba7d4847c4901dcd119b6bd6bb in term 1.
I20260806 02:58:49.552790 1611 leader_election.cc:302] T e629f8104e324abbbcea3b2a8022f973 P c2d58cba7d4847c4901dcd119b6bd6bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 451b2658f7d946c8a4cc453e96d8613f, c2d58cba7d4847c4901dcd119b6bd6bb; no voters:
I20260806 02:58:49.553145 1966 raft_consensus.cc:2781] T e629f8104e324abbbcea3b2a8022f973 P c2d58cba7d4847c4901dcd119b6bd6bb [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:58:49.553309 1966 raft_consensus.cc:686] T e629f8104e324abbbcea3b2a8022f973 P c2d58cba7d4847c4901dcd119b6bd6bb [term 1 LEADER]: Becoming Leader. State: Replica: c2d58cba7d4847c4901dcd119b6bd6bb, State: Running, Role: LEADER
I20260806 02:58:49.553620 1966 consensus_queue.cc:237] T e629f8104e324abbbcea3b2a8022f973 P c2d58cba7d4847c4901dcd119b6bd6bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451b2658f7d946c8a4cc453e96d8613f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37965 } } peers { permanent_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 36655 } } peers { permanent_uuid: "f0ac4bacaea845ad987b992b68df2772" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 39003 } }
I20260806 02:58:49.556981 1378 catalog_manager.cc:5495] T e629f8104e324abbbcea3b2a8022f973 P c2d58cba7d4847c4901dcd119b6bd6bb reported cstate change: term changed from 0 to 1, leader changed from <none> to c2d58cba7d4847c4901dcd119b6bd6bb (127.31.74.194). New cstate: current_term: 1 leader_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451b2658f7d946c8a4cc453e96d8613f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 36655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0ac4bacaea845ad987b992b68df2772" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 39003 } health_report { overall_health: UNKNOWN } } }
I20260806 02:58:49.645110 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:49.636332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55720 (local address 127.31.74.194:36655)
0806 02:58:49.637013 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:49.637018 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:49.637035 (+ 17us) server_negotiation.cc:407] Connection header received
0806 02:58:49.637103 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:49.637158 (+ 55us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:49.637232 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:49.637329 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:58:49.638234 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:49.640602 (+ 2368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:49.642509 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:49.643058 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:49.643300 (+ 242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:49.643867 (+ 567us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:58:49.643995 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:58:49.644092 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:49.644504 (+ 412us) server_negotiation.cc:299] Negotiation successful
0806 02:58:49.644663 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":430,"threads_started":1}
I20260806 02:58:49.658488 1821 raft_consensus.cc:1261] T e629f8104e324abbbcea3b2a8022f973 P f0ac4bacaea845ad987b992b68df2772 [term 1 FOLLOWER]: Refusing update from remote peer c2d58cba7d4847c4901dcd119b6bd6bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:58:49.658880 1537 raft_consensus.cc:1261] T e629f8104e324abbbcea3b2a8022f973 P 451b2658f7d946c8a4cc453e96d8613f [term 1 FOLLOWER]: Refusing update from remote peer c2d58cba7d4847c4901dcd119b6bd6bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:58:49.659654 1966 consensus_queue.cc:1035] T e629f8104e324abbbcea3b2a8022f973 P c2d58cba7d4847c4901dcd119b6bd6bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "451b2658f7d946c8a4cc453e96d8613f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:58:49.659989 1972 consensus_queue.cc:1035] T e629f8104e324abbbcea3b2a8022f973 P c2d58cba7d4847c4901dcd119b6bd6bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0ac4bacaea845ad987b992b68df2772" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 39003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:58:49.722712 1376 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51946:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:58:49.723026 1376 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51946:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:58:49.726383 1376 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 24ba9714cb1a494d983288e0ab1b4283: Sending DeleteTablet for 3 replicas of tablet e629f8104e324abbbcea3b2a8022f973
I20260806 02:58:49.727717 1517 tablet_service.cc:1514] Processing DeleteTablet for tablet e629f8104e324abbbcea3b2a8022f973 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:58:49 UTC) from {username='kudu', principal='kudu/127.31.74.254@KRBTEST.COM'} at 127.0.0.1:36948
I20260806 02:58:49.727716 1658 tablet_service.cc:1514] Processing DeleteTablet for tablet e629f8104e324abbbcea3b2a8022f973 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:58:49 UTC) from {username='kudu', principal='kudu/127.31.74.254@KRBTEST.COM'} at 127.0.0.1:37248
I20260806 02:58:49.727797 1802 tablet_service.cc:1514] Processing DeleteTablet for tablet e629f8104e324abbbcea3b2a8022f973 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:58:49 UTC) from {username='kudu', principal='kudu/127.31.74.254@KRBTEST.COM'} at 127.0.0.1:60902
I20260806 02:58:49.732147 1376 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51946:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:58:49.733598 1376 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:58:49.746405 1659 tablet_service.cc:1467] Processing CreateTablet for tablet be2974d4a71647cfb1ad579f60149fb9 (DEFAULT_TABLE table=test-table [id=09afc57ffe6f436c99ef303855af4e97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:58:49.746413 1801 tablet_service.cc:1467] Processing CreateTablet for tablet be2974d4a71647cfb1ad579f60149fb9 (DEFAULT_TABLE table=test-table [id=09afc57ffe6f436c99ef303855af4e97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:58:49.747107 1801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be2974d4a71647cfb1ad579f60149fb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:58:49.747102 1659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be2974d4a71647cfb1ad579f60149fb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:58:49.751065 2019 tablet_replica.cc:330] T e629f8104e324abbbcea3b2a8022f973 P f0ac4bacaea845ad987b992b68df2772: stopping tablet replica
I20260806 02:58:49.751286 2019 raft_consensus.cc:2229] T e629f8104e324abbbcea3b2a8022f973 P f0ac4bacaea845ad987b992b68df2772 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:58:49.751571 2019 raft_consensus.cc:2258] T e629f8104e324abbbcea3b2a8022f973 P f0ac4bacaea845ad987b992b68df2772 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:58:49.751806 1516 tablet_service.cc:1467] Processing CreateTablet for tablet be2974d4a71647cfb1ad579f60149fb9 (DEFAULT_TABLE table=test-table [id=09afc57ffe6f436c99ef303855af4e97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:58:49.752524 1516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be2974d4a71647cfb1ad579f60149fb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:58:49.752882 2019 ts_tablet_manager.cc:1905] T e629f8104e324abbbcea3b2a8022f973 P f0ac4bacaea845ad987b992b68df2772: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:58:49.758267 1995 tablet_bootstrap.cc:492] T be2974d4a71647cfb1ad579f60149fb9 P c2d58cba7d4847c4901dcd119b6bd6bb: Bootstrap starting.
I20260806 02:58:49.759150 2018 tablet_replica.cc:330] T e629f8104e324abbbcea3b2a8022f973 P 451b2658f7d946c8a4cc453e96d8613f: stopping tablet replica
I20260806 02:58:49.759737 2018 raft_consensus.cc:2229] T e629f8104e324abbbcea3b2a8022f973 P 451b2658f7d946c8a4cc453e96d8613f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:58:49.760095 2018 raft_consensus.cc:2258] T e629f8104e324abbbcea3b2a8022f973 P 451b2658f7d946c8a4cc453e96d8613f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:58:49.764717 2020 tablet_bootstrap.cc:492] T be2974d4a71647cfb1ad579f60149fb9 P f0ac4bacaea845ad987b992b68df2772: Bootstrap starting.
I20260806 02:58:49.765904 2018 ts_tablet_manager.cc:1905] T e629f8104e324abbbcea3b2a8022f973 P 451b2658f7d946c8a4cc453e96d8613f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:58:49.767652 2020 tablet_bootstrap.cc:654] T be2974d4a71647cfb1ad579f60149fb9 P f0ac4bacaea845ad987b992b68df2772: Neither blocks nor log segments found. Creating new log.
I20260806 02:58:49.771759 2020 tablet_bootstrap.cc:492] T be2974d4a71647cfb1ad579f60149fb9 P f0ac4bacaea845ad987b992b68df2772: No bootstrap required, opened a new log
I20260806 02:58:49.772078 2020 ts_tablet_manager.cc:1397] T be2974d4a71647cfb1ad579f60149fb9 P f0ac4bacaea845ad987b992b68df2772: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260806 02:58:49.773140 2020 raft_consensus.cc:348] T be2974d4a71647cfb1ad579f60149fb9 P f0ac4bacaea845ad987b992b68df2772 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451b2658f7d946c8a4cc453e96d8613f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37965 } } peers { permanent_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 36655 } } peers { permanent_uuid: "f0ac4bacaea845ad987b992b68df2772" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 39003 } }
I20260806 02:58:49.773524 2020 raft_consensus.cc:374] T be2974d4a71647cfb1ad579f60149fb9 P f0ac4bacaea845ad987b992b68df2772 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:58:49.773706 2020 raft_consensus.cc:729] T be2974d4a71647cfb1ad579f60149fb9 P f0ac4bacaea845ad987b992b68df2772 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0ac4bacaea845ad987b992b68df2772, State: Initialized, Role: FOLLOWER
I20260806 02:58:49.774171 2017 tablet_replica.cc:330] T e629f8104e324abbbcea3b2a8022f973 P c2d58cba7d4847c4901dcd119b6bd6bb: stopping tablet replica
I20260806 02:58:49.774204 1995 tablet_bootstrap.cc:654] T be2974d4a71647cfb1ad579f60149fb9 P c2d58cba7d4847c4901dcd119b6bd6bb: Neither blocks nor log segments found. Creating new log.
I20260806 02:58:49.774181 2020 consensus_queue.cc:260] T be2974d4a71647cfb1ad579f60149fb9 P f0ac4bacaea845ad987b992b68df2772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451b2658f7d946c8a4cc453e96d8613f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37965 } } peers { permanent_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 36655 } } peers { permanent_uuid: "f0ac4bacaea845ad987b992b68df2772" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 39003 } }
I20260806 02:58:49.774458 2017 raft_consensus.cc:2229] T e629f8104e324abbbcea3b2a8022f973 P c2d58cba7d4847c4901dcd119b6bd6bb [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:58:49.776969 2020 ts_tablet_manager.cc:1428] T be2974d4a71647cfb1ad579f60149fb9 P f0ac4bacaea845ad987b992b68df2772: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260806 02:58:49.777680 2017 raft_consensus.cc:2258] T e629f8104e324abbbcea3b2a8022f973 P c2d58cba7d4847c4901dcd119b6bd6bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:58:49.780210 2017 ts_tablet_manager.cc:1905] T e629f8104e324abbbcea3b2a8022f973 P c2d58cba7d4847c4901dcd119b6bd6bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:58:49.783128 2019 ts_tablet_manager.cc:1918] T e629f8104e324abbbcea3b2a8022f973 P f0ac4bacaea845ad987b992b68df2772: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:58:49.783324 2019 log.cc:1196] T e629f8104e324abbbcea3b2a8022f973 P f0ac4bacaea845ad987b992b68df2772: Deleting WAL directory at /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/minicluster-data/ts-2/wal/wals/e629f8104e324abbbcea3b2a8022f973
I20260806 02:58:49.783872 2019 ts_tablet_manager.cc:1939] T e629f8104e324abbbcea3b2a8022f973 P f0ac4bacaea845ad987b992b68df2772: Deleting consensus metadata
I20260806 02:58:49.784016 2022 tablet_bootstrap.cc:492] T be2974d4a71647cfb1ad579f60149fb9 P 451b2658f7d946c8a4cc453e96d8613f: Bootstrap starting.
I20260806 02:58:49.785939 2018 ts_tablet_manager.cc:1918] T e629f8104e324abbbcea3b2a8022f973 P 451b2658f7d946c8a4cc453e96d8613f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:58:49.786120 2018 log.cc:1196] T e629f8104e324abbbcea3b2a8022f973 P 451b2658f7d946c8a4cc453e96d8613f: Deleting WAL directory at /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/minicluster-data/ts-0/wal/wals/e629f8104e324abbbcea3b2a8022f973
I20260806 02:58:49.786705 2018 ts_tablet_manager.cc:1939] T e629f8104e324abbbcea3b2a8022f973 P 451b2658f7d946c8a4cc453e96d8613f: Deleting consensus metadata
I20260806 02:58:49.787127 2022 tablet_bootstrap.cc:654] T be2974d4a71647cfb1ad579f60149fb9 P 451b2658f7d946c8a4cc453e96d8613f: Neither blocks nor log segments found. Creating new log.
I20260806 02:58:49.787971 1364 catalog_manager.cc:4841] TS f0ac4bacaea845ad987b992b68df2772 (127.31.74.195:39003): tablet e629f8104e324abbbcea3b2a8022f973 (table test-table [id=4e11796a27454b1f92e2b04b28290f4e]) successfully deleted
I20260806 02:58:49.787998 1365 catalog_manager.cc:4841] TS 451b2658f7d946c8a4cc453e96d8613f (127.31.74.193:37965): tablet e629f8104e324abbbcea3b2a8022f973 (table test-table [id=4e11796a27454b1f92e2b04b28290f4e]) successfully deleted
I20260806 02:58:49.790124 1995 tablet_bootstrap.cc:492] T be2974d4a71647cfb1ad579f60149fb9 P c2d58cba7d4847c4901dcd119b6bd6bb: No bootstrap required, opened a new log
I20260806 02:58:49.790372 1995 ts_tablet_manager.cc:1397] T be2974d4a71647cfb1ad579f60149fb9 P c2d58cba7d4847c4901dcd119b6bd6bb: Time spent bootstrapping tablet: real 0.032s user 0.012s sys 0.004s
I20260806 02:58:49.791793 2022 tablet_bootstrap.cc:492] T be2974d4a71647cfb1ad579f60149fb9 P 451b2658f7d946c8a4cc453e96d8613f: No bootstrap required, opened a new log
I20260806 02:58:49.791810 1995 raft_consensus.cc:348] T be2974d4a71647cfb1ad579f60149fb9 P c2d58cba7d4847c4901dcd119b6bd6bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451b2658f7d946c8a4cc453e96d8613f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37965 } } peers { permanent_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 36655 } } peers { permanent_uuid: "f0ac4bacaea845ad987b992b68df2772" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 39003 } }
I20260806 02:58:49.792003 2022 ts_tablet_manager.cc:1397] T be2974d4a71647cfb1ad579f60149fb9 P 451b2658f7d946c8a4cc453e96d8613f: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20260806 02:58:49.792156 1995 raft_consensus.cc:374] T be2974d4a71647cfb1ad579f60149fb9 P c2d58cba7d4847c4901dcd119b6bd6bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:58:49.792284 1995 raft_consensus.cc:729] T be2974d4a71647cfb1ad579f60149fb9 P c2d58cba7d4847c4901dcd119b6bd6bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2d58cba7d4847c4901dcd119b6bd6bb, State: Initialized, Role: FOLLOWER
I20260806 02:58:49.792732 1995 consensus_queue.cc:260] T be2974d4a71647cfb1ad579f60149fb9 P c2d58cba7d4847c4901dcd119b6bd6bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451b2658f7d946c8a4cc453e96d8613f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37965 } } peers { permanent_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 36655 } } peers { permanent_uuid: "f0ac4bacaea845ad987b992b68df2772" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 39003 } }
I20260806 02:58:49.794289 1995 ts_tablet_manager.cc:1428] T be2974d4a71647cfb1ad579f60149fb9 P c2d58cba7d4847c4901dcd119b6bd6bb: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260806 02:58:49.795187 2022 raft_consensus.cc:348] T be2974d4a71647cfb1ad579f60149fb9 P 451b2658f7d946c8a4cc453e96d8613f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451b2658f7d946c8a4cc453e96d8613f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37965 } } peers { permanent_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 36655 } } peers { permanent_uuid: "f0ac4bacaea845ad987b992b68df2772" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 39003 } }
I20260806 02:58:49.795490 2022 raft_consensus.cc:374] T be2974d4a71647cfb1ad579f60149fb9 P 451b2658f7d946c8a4cc453e96d8613f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:58:49.795652 2022 raft_consensus.cc:729] T be2974d4a71647cfb1ad579f60149fb9 P 451b2658f7d946c8a4cc453e96d8613f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 451b2658f7d946c8a4cc453e96d8613f, State: Initialized, Role: FOLLOWER
I20260806 02:58:49.795977 2022 consensus_queue.cc:260] T be2974d4a71647cfb1ad579f60149fb9 P 451b2658f7d946c8a4cc453e96d8613f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451b2658f7d946c8a4cc453e96d8613f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37965 } } peers { permanent_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 36655 } } peers { permanent_uuid: "f0ac4bacaea845ad987b992b68df2772" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 39003 } }
I20260806 02:58:49.796777 2022 ts_tablet_manager.cc:1428] T be2974d4a71647cfb1ad579f60149fb9 P 451b2658f7d946c8a4cc453e96d8613f: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20260806 02:58:49.798640 2017 ts_tablet_manager.cc:1918] T e629f8104e324abbbcea3b2a8022f973 P c2d58cba7d4847c4901dcd119b6bd6bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:58:49.798830 2017 log.cc:1196] T e629f8104e324abbbcea3b2a8022f973 P c2d58cba7d4847c4901dcd119b6bd6bb: Deleting WAL directory at /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/minicluster-data/ts-1/wal/wals/e629f8104e324abbbcea3b2a8022f973
I20260806 02:58:49.799351 2017 ts_tablet_manager.cc:1939] T e629f8104e324abbbcea3b2a8022f973 P c2d58cba7d4847c4901dcd119b6bd6bb: Deleting consensus metadata
I20260806 02:58:49.801106 1363 catalog_manager.cc:4841] TS c2d58cba7d4847c4901dcd119b6bd6bb (127.31.74.194:36655): tablet e629f8104e324abbbcea3b2a8022f973 (table test-table [id=4e11796a27454b1f92e2b04b28290f4e]) successfully deleted
I20260806 02:58:49.820907 1999 raft_consensus.cc:482] T be2974d4a71647cfb1ad579f60149fb9 P f0ac4bacaea845ad987b992b68df2772 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:58:49.821147 1999 raft_consensus.cc:504] T be2974d4a71647cfb1ad579f60149fb9 P f0ac4bacaea845ad987b992b68df2772 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451b2658f7d946c8a4cc453e96d8613f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37965 } } peers { permanent_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 36655 } } peers { permanent_uuid: "f0ac4bacaea845ad987b992b68df2772" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 39003 } }
I20260806 02:58:49.823086 1999 leader_election.cc:288] T be2974d4a71647cfb1ad579f60149fb9 P f0ac4bacaea845ad987b992b68df2772 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 451b2658f7d946c8a4cc453e96d8613f (127.31.74.193:37965), c2d58cba7d4847c4901dcd119b6bd6bb (127.31.74.194:36655)
I20260806 02:58:49.824054 1536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be2974d4a71647cfb1ad579f60149fb9" candidate_uuid: "f0ac4bacaea845ad987b992b68df2772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "451b2658f7d946c8a4cc453e96d8613f" is_pre_election: true
I20260806 02:58:49.824067 1679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be2974d4a71647cfb1ad579f60149fb9" candidate_uuid: "f0ac4bacaea845ad987b992b68df2772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" is_pre_election: true
I20260806 02:58:49.824404 1536 raft_consensus.cc:2454] T be2974d4a71647cfb1ad579f60149fb9 P 451b2658f7d946c8a4cc453e96d8613f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0ac4bacaea845ad987b992b68df2772 in term 0.
I20260806 02:58:49.824406 1679 raft_consensus.cc:2454] T be2974d4a71647cfb1ad579f60149fb9 P c2d58cba7d4847c4901dcd119b6bd6bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0ac4bacaea845ad987b992b68df2772 in term 0.
I20260806 02:58:49.825184 1755 leader_election.cc:302] T be2974d4a71647cfb1ad579f60149fb9 P f0ac4bacaea845ad987b992b68df2772 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 451b2658f7d946c8a4cc453e96d8613f, c2d58cba7d4847c4901dcd119b6bd6bb, f0ac4bacaea845ad987b992b68df2772; no voters:
I20260806 02:58:49.825591 1999 raft_consensus.cc:2781] T be2974d4a71647cfb1ad579f60149fb9 P f0ac4bacaea845ad987b992b68df2772 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:58:49.825723 1999 raft_consensus.cc:482] T be2974d4a71647cfb1ad579f60149fb9 P f0ac4bacaea845ad987b992b68df2772 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:58:49.825856 1999 raft_consensus.cc:3037] T be2974d4a71647cfb1ad579f60149fb9 P f0ac4bacaea845ad987b992b68df2772 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:58:49.828783 1999 raft_consensus.cc:504] T be2974d4a71647cfb1ad579f60149fb9 P f0ac4bacaea845ad987b992b68df2772 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451b2658f7d946c8a4cc453e96d8613f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37965 } } peers { permanent_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 36655 } } peers { permanent_uuid: "f0ac4bacaea845ad987b992b68df2772" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 39003 } }
I20260806 02:58:49.829972 1999 leader_election.cc:288] T be2974d4a71647cfb1ad579f60149fb9 P f0ac4bacaea845ad987b992b68df2772 [CANDIDATE]: Term 1 election: Requested vote from peers 451b2658f7d946c8a4cc453e96d8613f (127.31.74.193:37965), c2d58cba7d4847c4901dcd119b6bd6bb (127.31.74.194:36655)
I20260806 02:58:49.830315 1536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be2974d4a71647cfb1ad579f60149fb9" candidate_uuid: "f0ac4bacaea845ad987b992b68df2772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "451b2658f7d946c8a4cc453e96d8613f"
I20260806 02:58:49.830642 1536 raft_consensus.cc:3037] T be2974d4a71647cfb1ad579f60149fb9 P 451b2658f7d946c8a4cc453e96d8613f [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:58:49.831066 1679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be2974d4a71647cfb1ad579f60149fb9" candidate_uuid: "f0ac4bacaea845ad987b992b68df2772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb"
I20260806 02:58:49.831637 1679 raft_consensus.cc:3037] T be2974d4a71647cfb1ad579f60149fb9 P c2d58cba7d4847c4901dcd119b6bd6bb [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:58:49.833380 1536 raft_consensus.cc:2454] T be2974d4a71647cfb1ad579f60149fb9 P 451b2658f7d946c8a4cc453e96d8613f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0ac4bacaea845ad987b992b68df2772 in term 1.
I20260806 02:58:49.834129 1755 leader_election.cc:302] T be2974d4a71647cfb1ad579f60149fb9 P f0ac4bacaea845ad987b992b68df2772 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 451b2658f7d946c8a4cc453e96d8613f, f0ac4bacaea845ad987b992b68df2772; no voters:
I20260806 02:58:49.834240 1679 raft_consensus.cc:2454] T be2974d4a71647cfb1ad579f60149fb9 P c2d58cba7d4847c4901dcd119b6bd6bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0ac4bacaea845ad987b992b68df2772 in term 1.
I20260806 02:58:49.835299 1999 raft_consensus.cc:2781] T be2974d4a71647cfb1ad579f60149fb9 P f0ac4bacaea845ad987b992b68df2772 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:58:49.835865 1999 raft_consensus.cc:686] T be2974d4a71647cfb1ad579f60149fb9 P f0ac4bacaea845ad987b992b68df2772 [term 1 LEADER]: Becoming Leader. State: Replica: f0ac4bacaea845ad987b992b68df2772, State: Running, Role: LEADER
I20260806 02:58:49.836407 1999 consensus_queue.cc:237] T be2974d4a71647cfb1ad579f60149fb9 P f0ac4bacaea845ad987b992b68df2772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451b2658f7d946c8a4cc453e96d8613f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37965 } } peers { permanent_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 36655 } } peers { permanent_uuid: "f0ac4bacaea845ad987b992b68df2772" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 39003 } }
I20260806 02:58:49.841642 1377 catalog_manager.cc:5495] T be2974d4a71647cfb1ad579f60149fb9 P f0ac4bacaea845ad987b992b68df2772 reported cstate change: term changed from 0 to 1, leader changed from <none> to f0ac4bacaea845ad987b992b68df2772 (127.31.74.195). New cstate: current_term: 1 leader_uuid: "f0ac4bacaea845ad987b992b68df2772" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451b2658f7d946c8a4cc453e96d8613f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 36655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0ac4bacaea845ad987b992b68df2772" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 39003 } health_report { overall_health: HEALTHY } } }
I20260806 02:58:49.886529 2027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:49.877437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34806 (local address 127.31.74.195:39003)
0806 02:58:49.878115 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:49.878120 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:49.878143 (+ 23us) server_negotiation.cc:407] Connection header received
0806 02:58:49.878244 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:49.878247 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:49.878332 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:49.878521 (+ 189us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:58:49.879109 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:49.879967 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:49.882775 (+ 2808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:49.883334 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:49.883661 (+ 327us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:49.884573 (+ 912us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:58:49.885402 (+ 829us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:58:49.885543 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:49.885962 (+ 419us) server_negotiation.cc:299] Negotiation successful
0806 02:58:49.886134 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":406,"threads_started":1}
I20260806 02:58:49.912412 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:49.904197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.195:39657 (local address 127.31.74.194:36655)
0806 02:58:49.904868 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:49.904875 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:49.905182 (+ 307us) server_negotiation.cc:407] Connection header received
0806 02:58:49.905447 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:49.905451 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:49.905525 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:49.905636 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:58:49.906179 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:49.907298 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:49.910009 (+ 2711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:49.910501 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:49.910716 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:49.911224 (+ 508us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:58:49.911371 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:58:49.911487 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:49.911887 (+ 400us) server_negotiation.cc:299] Negotiation successful
0806 02:58:49.911988 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495}
I20260806 02:58:49.912503 2030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:49.903848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.194:36655 (local address 127.31.74.195:39657)
0806 02:58:49.905045 (+ 1197us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:49.905063 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:58:49.905283 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:58:49.905658 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:58:49.905663 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:58:49.905686 (+ 23us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 02:58:49.906010 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:49.906019 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:49.907930 (+ 1911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:49.907936 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:49.909779 (+ 1843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:49.909798 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:49.910753 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:49.910760 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:49.910936 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:49.911008 (+ 72us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 02:58:49.911488 (+ 480us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 02:58:49.911505 (+ 17us) client_negotiation.cc:769] Sending connection context
0806 02:58:49.911866 (+ 361us) client_negotiation.cc:240] Negotiation successful
0806 02:58:49.912048 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1020,"thread_start_us":381,"threads_started":1}
I20260806 02:58:49.926443 1679 raft_consensus.cc:1261] T be2974d4a71647cfb1ad579f60149fb9 P c2d58cba7d4847c4901dcd119b6bd6bb [term 1 FOLLOWER]: Refusing update from remote peer f0ac4bacaea845ad987b992b68df2772: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:58:49.926529 1536 raft_consensus.cc:1261] T be2974d4a71647cfb1ad579f60149fb9 P 451b2658f7d946c8a4cc453e96d8613f [term 1 FOLLOWER]: Refusing update from remote peer f0ac4bacaea845ad987b992b68df2772: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:58:49.927558 2025 consensus_queue.cc:1035] T be2974d4a71647cfb1ad579f60149fb9 P f0ac4bacaea845ad987b992b68df2772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2d58cba7d4847c4901dcd119b6bd6bb" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 36655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:58:49.928550 1999 consensus_queue.cc:1035] T be2974d4a71647cfb1ad579f60149fb9 P f0ac4bacaea845ad987b992b68df2772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "451b2658f7d946c8a4cc453e96d8613f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:58:49.934649 2006 mvcc.cc:205] Tried to move back new op lower bound from 7315395092168785920 to 7315395091815682048. Current Snapshot: MvccSnapshot[applied={T|T < 7315395092168785920}]
I20260806 02:58:49.941776 2005 mvcc.cc:205] Tried to move back new op lower bound from 7315395092168785920 to 7315395091815682048. Current Snapshot: MvccSnapshot[applied={T|T < 7315395092168785920}]
I20260806 02:58:49.975180 2027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:49.961140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.194:45517 (local address 127.31.74.195:39003)
0806 02:58:49.961416 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:49.961426 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:49.962137 (+ 711us) server_negotiation.cc:407] Connection header received
0806 02:58:49.962524 (+ 387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:49.962530 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:49.962700 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:49.963703 (+ 1003us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:58:49.965773 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:49.968555 (+ 2782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:49.971717 (+ 3162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:49.972455 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:49.972629 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:49.973843 (+ 1214us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:58:49.974032 (+ 189us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:58:49.974154 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:49.974689 (+ 535us) server_negotiation.cc:299] Negotiation successful
0806 02:58:49.974802 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260806 02:58:49.975550 2034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:49.961140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.195:39003 (local address 127.31.74.194:45517)
0806 02:58:49.962007 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:49.962026 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 02:58:49.962217 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:58:49.963722 (+ 1505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:58:49.963727 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:58:49.963743 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 02:58:49.965387 (+ 1644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:49.965462 (+ 75us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:49.968867 (+ 3405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:49.968878 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:49.971365 (+ 2487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:49.971405 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:49.973086 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:49.973099 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:49.973532 (+ 433us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:49.973611 (+ 79us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 02:58:49.974217 (+ 606us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 02:58:49.974330 (+ 113us) client_negotiation.cc:769] Sending connection context
0806 02:58:49.974468 (+ 138us) client_negotiation.cc:240] Negotiation successful
0806 02:58:49.974703 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":698,"thread_start_us":522,"threads_started":1}
W20260806 02:58:49.977218 2006 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260806 02:58:51.608536 1376 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51946:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:58:51.608793 1376 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51946:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:58:51.615658 1376 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 24ba9714cb1a494d983288e0ab1b4283: Sending DeleteTablet for 3 replicas of tablet be2974d4a71647cfb1ad579f60149fb9
I20260806 02:58:51.617157 1516 tablet_service.cc:1514] Processing DeleteTablet for tablet be2974d4a71647cfb1ad579f60149fb9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:58:51 UTC) from {username='kudu', principal='kudu/127.31.74.254@KRBTEST.COM'} at 127.0.0.1:36948
I20260806 02:58:51.618561 1658 tablet_service.cc:1514] Processing DeleteTablet for tablet be2974d4a71647cfb1ad579f60149fb9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:58:51 UTC) from {username='kudu', principal='kudu/127.31.74.254@KRBTEST.COM'} at 127.0.0.1:37248
I20260806 02:58:51.618561 1801 tablet_service.cc:1514] Processing DeleteTablet for tablet be2974d4a71647cfb1ad579f60149fb9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:58:51 UTC) from {username='kudu', principal='kudu/127.31.74.254@KRBTEST.COM'} at 127.0.0.1:60902
I20260806 02:58:51.620069 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 1450
I20260806 02:58:51.629914 2063 tablet_replica.cc:330] T be2974d4a71647cfb1ad579f60149fb9 P c2d58cba7d4847c4901dcd119b6bd6bb: stopping tablet replica
I20260806 02:58:51.629969 2062 tablet_replica.cc:330] T be2974d4a71647cfb1ad579f60149fb9 P f0ac4bacaea845ad987b992b68df2772: stopping tablet replica
I20260806 02:58:51.630198 2063 raft_consensus.cc:2229] T be2974d4a71647cfb1ad579f60149fb9 P c2d58cba7d4847c4901dcd119b6bd6bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:58:51.630486 2063 raft_consensus.cc:2258] T be2974d4a71647cfb1ad579f60149fb9 P c2d58cba7d4847c4901dcd119b6bd6bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:58:51.630465 2062 raft_consensus.cc:2229] T be2974d4a71647cfb1ad579f60149fb9 P f0ac4bacaea845ad987b992b68df2772 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:58:51.631319 2062 raft_consensus.cc:2258] T be2974d4a71647cfb1ad579f60149fb9 P f0ac4bacaea845ad987b992b68df2772 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:58:51.632304 2061 tablet_replica.cc:330] T be2974d4a71647cfb1ad579f60149fb9 P 451b2658f7d946c8a4cc453e96d8613f: stopping tablet replica
I20260806 02:58:51.632948 2061 raft_consensus.cc:2229] T be2974d4a71647cfb1ad579f60149fb9 P 451b2658f7d946c8a4cc453e96d8613f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:58:51.634703 2062 ts_tablet_manager.cc:1905] T be2974d4a71647cfb1ad579f60149fb9 P f0ac4bacaea845ad987b992b68df2772: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:58:51.634644 2063 ts_tablet_manager.cc:1905] T be2974d4a71647cfb1ad579f60149fb9 P c2d58cba7d4847c4901dcd119b6bd6bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:58:51.634840 2061 raft_consensus.cc:2258] T be2974d4a71647cfb1ad579f60149fb9 P 451b2658f7d946c8a4cc453e96d8613f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:58:51.640611 2061 ts_tablet_manager.cc:1905] T be2974d4a71647cfb1ad579f60149fb9 P 451b2658f7d946c8a4cc453e96d8613f: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 06 02:58:51 dist-test-slave-dhph krb5kdc[1333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985129, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.74.193@KRBTEST.COM
I20260806 02:58:51.661082 2064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:51.624257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38848 (local address 127.31.74.193:37965)
0806 02:58:51.630526 (+ 6269us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:51.630538 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:51.631654 (+ 1116us) server_negotiation.cc:407] Connection header received
0806 02:58:51.631736 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:51.631741 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:51.631848 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:51.631992 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:51.635384 (+ 3392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:51.639448 (+ 4064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:51.643114 (+ 3666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:51.643670 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:51.643833 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:51.648650 (+ 4817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:58:51.648670 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:58:51.648687 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:58:51.648732 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:58:51.654754 (+ 6022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:51.656394 (+ 1640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:51.656404 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:51.656407 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:51.656606 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:51.658349 (+ 1743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:51.658354 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:51.658356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:51.659380 (+ 1024us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:58:51.659742 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:51.660229 (+ 487us) server_negotiation.cc:299] Negotiation successful
0806 02:58:51.660735 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5925,"thread_start_us":5743,"threads_started":1}
W20260806 02:58:51.662249 1583 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38848
W20260806 02:58:51.663219 32043 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 1450: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 02:58:51.663476 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 1450
I20260806 02:58:51.693289 2062 ts_tablet_manager.cc:1918] T be2974d4a71647cfb1ad579f60149fb9 P f0ac4bacaea845ad987b992b68df2772: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 02:58:51.693608 2062 log.cc:1196] T be2974d4a71647cfb1ad579f60149fb9 P f0ac4bacaea845ad987b992b68df2772: Deleting WAL directory at /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/minicluster-data/ts-2/wal/wals/be2974d4a71647cfb1ad579f60149fb9
I20260806 02:58:51.693397 2063 ts_tablet_manager.cc:1918] T be2974d4a71647cfb1ad579f60149fb9 P c2d58cba7d4847c4901dcd119b6bd6bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 02:58:51.693938 2063 log.cc:1196] T be2974d4a71647cfb1ad579f60149fb9 P c2d58cba7d4847c4901dcd119b6bd6bb: Deleting WAL directory at /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785985105255475-32043-0/minicluster-data/ts-1/wal/wals/be2974d4a71647cfb1ad579f60149fb9
I20260806 02:58:51.694468 2063 ts_tablet_manager.cc:1939] T be2974d4a71647cfb1ad579f60149fb9 P c2d58cba7d4847c4901dcd119b6bd6bb: Deleting consensus metadata
I20260806 02:58:51.694923 2062 ts_tablet_manager.cc:1939] T be2974d4a71647cfb1ad579f60149fb9 P f0ac4bacaea845ad987b992b68df2772: Deleting consensus metadata
W20260806 02:58:51.724754 1365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.74.193:37965 (error 108)
I20260806 02:58:51.725337 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 1592
I20260806 02:58:51.727985 2068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:51.726673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:37965 (local address 127.0.0.1:38858)
0806 02:58:51.727399 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:51.727664 (+ 265us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.193:37965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":341,"threads_started":1}
W20260806 02:58:51.728521 1365 catalog_manager.cc:4584] TS 451b2658f7d946c8a4cc453e96d8613f (127.31.74.193:37965): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet be2974d4a71647cfb1ad579f60149fb9: Network error: Client connection negotiation failed: client connection to 127.31.74.193:37965: connect: Connection refused (error 111)
Aug 06 02:58:51 dist-test-slave-dhph krb5kdc[1333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985129, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.74.194@KRBTEST.COM
I20260806 02:58:51.751403 2067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:51.726227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55726 (local address 127.31.74.194:36655)
0806 02:58:51.726837 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:51.726845 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:51.726866 (+ 21us) server_negotiation.cc:407] Connection header received
0806 02:58:51.726938 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:51.726945 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:51.727036 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:51.728371 (+ 1335us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:51.729253 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:51.730073 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:51.731954 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:51.732425 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:51.732917 (+ 492us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:51.742369 (+ 9452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:58:51.742390 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:58:51.742409 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:58:51.742458 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:58:51.747233 (+ 4775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:51.748556 (+ 1323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:51.748568 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:51.748571 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:51.748785 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:51.749268 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:51.749272 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:51.749274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:51.749666 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:58:51.750037 (+ 371us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:51.750538 (+ 501us) server_negotiation.cc:299] Negotiation successful
0806 02:58:51.750952 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":616,"threads_started":1}
W20260806 02:58:51.752450 1726 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55726
W20260806 02:58:51.754492 32043 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 1592: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 02:58:51.755311 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 1592
I20260806 02:58:51.762902 1364 catalog_manager.cc:4841] TS f0ac4bacaea845ad987b992b68df2772 (127.31.74.195:39003): tablet be2974d4a71647cfb1ad579f60149fb9 (table test-table [id=09afc57ffe6f436c99ef303855af4e97]) successfully deleted
I20260806 02:58:51.782989 2068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:51.782372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:37965 (local address 127.0.0.1:38870)
0806 02:58:51.782644 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:51.782770 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.193:37965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260806 02:58:51.784428 2068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:51.783844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:37965 (local address 127.0.0.1:38884)
0806 02:58:51.784044 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:51.784186 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.193:37965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260806 02:58:51.797462 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 1735
I20260806 02:58:51.799376 2068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:51.798434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.194:36655 (local address 127.0.0.1:55728)
0806 02:58:51.798712 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:51.799055 (+ 343us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.194:36655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
Aug 06 02:58:51 dist-test-slave-dhph krb5kdc[1333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985129, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.74.195@KRBTEST.COM
I20260806 02:58:51.824937 2069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:51.799319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34822 (local address 127.31.74.195:39003)
0806 02:58:51.800524 (+ 1205us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:51.800530 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:51.800558 (+ 28us) server_negotiation.cc:407] Connection header received
0806 02:58:51.800647 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:51.800651 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:51.800739 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:51.801044 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:51.803062 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:51.803908 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:51.805266 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:51.805764 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:51.805876 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:51.812815 (+ 6939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:58:51.812835 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:58:51.812853 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:58:51.812916 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:58:51.819324 (+ 6408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:51.821095 (+ 1771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:51.821111 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:51.821114 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:51.821422 (+ 308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:51.822939 (+ 1517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:51.822946 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:51.822948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:51.823550 (+ 602us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:58:51.824131 (+ 581us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:51.824344 (+ 213us) server_negotiation.cc:299] Negotiation successful
0806 02:58:51.824603 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":913,"thread_start_us":719,"threads_started":1}
W20260806 02:58:51.825898 1868 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34822
W20260806 02:58:51.826998 32043 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 1735: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 02:58:51.827195 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 1735
I20260806 02:58:51.833060 2068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:51.831677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.194:36655 (local address 127.0.0.1:55734)
0806 02:58:51.831970 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:51.832097 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.194:36655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260806 02:58:51.834673 2068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:51.834044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.194:36655 (local address 127.0.0.1:55742)
0806 02:58:51.834320 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:51.834446 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.194:36655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260806 02:58:51.845947 2068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:51.845185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:37965 (local address 127.0.0.1:38898)
0806 02:58:51.845545 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:51.845680 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.193:37965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260806 02:58:51.848074 2068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:51.847482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:37965 (local address 127.0.0.1:38910)
0806 02:58:51.847704 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:51.847887 (+ 183us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.193:37965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260806 02:58:51.869969 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 1346
I20260806 02:58:51.871604 1418 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:58:52.046060 2068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:52.044804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.194:36655 (local address 127.0.0.1:55756)
0806 02:58:52.045632 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:52.045727 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.194:36655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":672}
I20260806 02:58:52.047369 2068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:52.045826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:37965 (local address 127.0.0.1:38912)
0806 02:58:52.046635 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:52.046780 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.193:37965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456,"threads_started":1}
I20260806 02:58:52.048475 2068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:52.047561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.194:36655 (local address 127.0.0.1:55764)
0806 02:58:52.048093 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:52.048284 (+ 191us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.194:36655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260806 02:58:52.048866 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 1346
2026-08-06T02:58:52Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (10520 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:58:52 dist-test-slave-dhph krb5kdc[2074](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:58:52 dist-test-slave-dhph krb5kdc[2074](info): set up 2 sockets
Aug 06 02:58:52 dist-test-slave-dhph krb5kdc[2074](info): commencing operation
krb5kdc: starting...
W20260806 02:58:53.667665 32043 mini_kdc.cc:121] Time spent starting KDC: real 1.545s user 0.000s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/krb5kdc/test-user.keytab.
Aug 06 02:58:53 dist-test-slave-dhph krb5kdc[2074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985133, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:58:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:58:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.74.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab.
Entry for principal kudu/127.31.74.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab.
I20260806 02:58:53.956756 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:43945
--webserver_interface=127.31.74.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab
--principal=kudu/127.31.74.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:43477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.74.254:43945
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:58:54.365080 2087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:58:54.365485 2087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:58:54.365664 2087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:58:54.375013 2087 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:58:54.375240 2087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:58:54.375375 2087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:58:54.375506 2087 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:58:54.375602 2087 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:58:54.388091 2087 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:43477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.74.254:43945
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab
--principal=kudu/127.31.74.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:43945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.74.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:58:54.389931 2087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:58:54.391737 2087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:58:54.406674 2093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:58:54.406987 2092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:58:54.407027 2095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:58:54.407550 2087 server_base.cc:1034] running on GCE node
I20260806 02:58:54.408712 2087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:58:54.410404 2087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:58:54.411746 2087 hybrid_clock.cc:648] HybridClock initialized: now 1785985134411677 us; error 54 us; skew 500 ppm
I20260806 02:58:54.412326 2087 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:58:54 dist-test-slave-dhph krb5kdc[2074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985134, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:58:54.421900 2087 init.cc:377] Logged in from keytab as kudu/127.31.74.254@KRBTEST.COM (short username kudu)
I20260806 02:58:54.423755 2087 webserver.cc:460] Webserver started at http://127.31.74.254:35083/ using document root <none> and password file <none>
I20260806 02:58:54.424530 2087 fs_manager.cc:359] Metadata directory not provided
I20260806 02:58:54.424681 2087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:58:54.425001 2087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:58:54.428671 2087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/minicluster-data/master-0/data/instance:
uuid: "5fc5a765dd574198b63dce332d0e45d7"
format_stamp: "Formatted at 2026-08-06 02:58:54 on dist-test-slave-dhph"
server_key: "66b1f13c35c9bf0dffc73085374139d5"
server_key_iv: "2fb74b7d0ba22715023a7602cad8aeed"
server_key_version: "encryptionkey@0"
I20260806 02:58:54.429549 2087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/minicluster-data/master-0/wal/instance:
uuid: "5fc5a765dd574198b63dce332d0e45d7"
format_stamp: "Formatted at 2026-08-06 02:58:54 on dist-test-slave-dhph"
server_key: "66b1f13c35c9bf0dffc73085374139d5"
server_key_iv: "2fb74b7d0ba22715023a7602cad8aeed"
server_key_version: "encryptionkey@0"
I20260806 02:58:54.435513 2087 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260806 02:58:54.441509 2102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:58:54.443498 2087 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.005s
I20260806 02:58:54.443754 2087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/minicluster-data/master-0/wal
uuid: "5fc5a765dd574198b63dce332d0e45d7"
format_stamp: "Formatted at 2026-08-06 02:58:54 on dist-test-slave-dhph"
server_key: "66b1f13c35c9bf0dffc73085374139d5"
server_key_iv: "2fb74b7d0ba22715023a7602cad8aeed"
server_key_version: "encryptionkey@0"
I20260806 02:58:54.443992 2087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:58:54.465649 2087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:58:54.470999 2087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:58:54.471342 2087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:58:54.492599 2087 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.254:43945
I20260806 02:58:54.492632 2154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.254:43945 every 8 connection(s)
I20260806 02:58:54.494055 2087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
I20260806 02:58:54.497012 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 2087
I20260806 02:58:54.497588 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/minicluster-data/master-0/wal/instance
I20260806 02:58:54.498462 32043 external_mini_cluster.cc:1428] Setting key 4c9bdb161fe39527d5ed1aaf1d6b13ff
I20260806 02:58:54.498464 2155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:58:54.516134 2155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fc5a765dd574198b63dce332d0e45d7: Bootstrap starting.
Aug 06 02:58:54 dist-test-slave-dhph krb5kdc[2074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985133, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 02:58:54.520804 2155 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5fc5a765dd574198b63dce332d0e45d7: Neither blocks nor log segments found. Creating new log.
I20260806 02:58:54.522070 2155 log.cc:824] T 00000000000000000000000000000000 P 5fc5a765dd574198b63dce332d0e45d7: Log is configured to *not* fsync() on all Append() calls
I20260806 02:58:54.525861 2155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fc5a765dd574198b63dce332d0e45d7: No bootstrap required, opened a new log
I20260806 02:58:54.530730 2158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:54.501192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38522 (local address 127.31.74.254:43945)
0806 02:58:54.502447 (+ 1255us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:54.502461 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:54.502507 (+ 46us) server_negotiation.cc:407] Connection header received
0806 02:58:54.504236 (+ 1729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:54.504261 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:54.504969 (+ 708us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:54.505366 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:54.507515 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:54.509107 (+ 1592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:54.511386 (+ 2279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:54.512241 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:54.512860 (+ 619us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:54.520490 (+ 7630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:58:54.520504 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:58:54.520530 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:58:54.520608 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:58:54.525019 (+ 4411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:54.526161 (+ 1142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:54.526169 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:54.526176 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:54.526442 (+ 266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:54.526942 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:54.526948 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:54.526950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:54.527802 (+ 852us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:58:54.528282 (+ 480us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:54.528847 (+ 565us) server_negotiation.cc:299] Negotiation successful
0806 02:58:54.529414 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":962,"thread_start_us":535,"threads_started":1}
I20260806 02:58:54.536546 2155 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5fc5a765dd574198b63dce332d0e45d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fc5a765dd574198b63dce332d0e45d7" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 43945 } }
I20260806 02:58:54.537029 2155 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5fc5a765dd574198b63dce332d0e45d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:58:54.537153 2155 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5fc5a765dd574198b63dce332d0e45d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fc5a765dd574198b63dce332d0e45d7, State: Initialized, Role: FOLLOWER
I20260806 02:58:54.537871 2155 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5fc5a765dd574198b63dce332d0e45d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fc5a765dd574198b63dce332d0e45d7" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 43945 } }
I20260806 02:58:54.538163 2155 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5fc5a765dd574198b63dce332d0e45d7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:58:54.538338 2155 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5fc5a765dd574198b63dce332d0e45d7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:58:54.538534 2155 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5fc5a765dd574198b63dce332d0e45d7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:58:54.541409 2155 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5fc5a765dd574198b63dce332d0e45d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fc5a765dd574198b63dce332d0e45d7" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 43945 } }
I20260806 02:58:54.542104 2155 leader_election.cc:302] T 00000000000000000000000000000000 P 5fc5a765dd574198b63dce332d0e45d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5fc5a765dd574198b63dce332d0e45d7; no voters:
I20260806 02:58:54.542902 2155 leader_election.cc:288] T 00000000000000000000000000000000 P 5fc5a765dd574198b63dce332d0e45d7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:58:54.543062 2160 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5fc5a765dd574198b63dce332d0e45d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:58:54.544086 2160 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5fc5a765dd574198b63dce332d0e45d7 [term 1 LEADER]: Becoming Leader. State: Replica: 5fc5a765dd574198b63dce332d0e45d7, State: Running, Role: LEADER
I20260806 02:58:54.544757 2160 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5fc5a765dd574198b63dce332d0e45d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fc5a765dd574198b63dce332d0e45d7" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 43945 } }
I20260806 02:58:54.545526 2155 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5fc5a765dd574198b63dce332d0e45d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:58:54.547842 2161 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fc5a765dd574198b63dce332d0e45d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5fc5a765dd574198b63dce332d0e45d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fc5a765dd574198b63dce332d0e45d7" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 43945 } } }
I20260806 02:58:54.547866 2162 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fc5a765dd574198b63dce332d0e45d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5fc5a765dd574198b63dce332d0e45d7. Latest consensus state: current_term: 1 leader_uuid: "5fc5a765dd574198b63dce332d0e45d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fc5a765dd574198b63dce332d0e45d7" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 43945 } } }
I20260806 02:58:54.549327 2162 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fc5a765dd574198b63dce332d0e45d7 [sys.catalog]: This master's current role is: LEADER
I20260806 02:58:54.549322 2161 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fc5a765dd574198b63dce332d0e45d7 [sys.catalog]: This master's current role is: LEADER
I20260806 02:58:54.551187 2169 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:58:54.558933 2169 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:58:54.570863 2169 catalog_manager.cc:1347] Generated new cluster ID: 67ead4962d21457bbc38a44e0b46790b
I20260806 02:58:54.571027 2169 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:58:54.597273 2169 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:58:54.599907 2169 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:58:54.612461 2169 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5fc5a765dd574198b63dce332d0e45d7: Generated new TSK 0
I20260806 02:58:54.613572 2169 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.74.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.193.keytab.
Entry for principal kudu/127.31.74.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.193.keytab.
I20260806 02:58:54.695802 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.193:0
--local_ip_for_outbound_sockets=127.31.74.193
--webserver_interface=127.31.74.193
--webserver_port=0
--tserver_master_addrs=127.31.74.254:43945
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.193.keytab
--principal=kudu/127.31.74.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:43477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:58:55.098441 2181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:58:55.098861 2181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:58:55.099040 2181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:58:55.106642 2181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:58:55.106786 2181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:58:55.106958 2181 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.193
I20260806 02:58:55.119391 2181 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:43477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.193.keytab
--principal=kudu/127.31.74.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.74.193
--webserver_port=0
--tserver_master_addrs=127.31.74.254:43945
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.193
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:58:55.120829 2181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:58:55.122290 2181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:58:55.139050 2187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:58:55.139101 2186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:58:55.139127 2189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:58:55.141101 2181 server_base.cc:1034] running on GCE node
I20260806 02:58:55.141891 2181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:58:55.143055 2181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:58:55.144685 2181 hybrid_clock.cc:648] HybridClock initialized: now 1785985135144625 us; error 38 us; skew 500 ppm
I20260806 02:58:55.145048 2181 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:58:55 dist-test-slave-dhph krb5kdc[2074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985135, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:58:55.149554 2181 init.cc:377] Logged in from keytab as kudu/127.31.74.193@KRBTEST.COM (short username kudu)
I20260806 02:58:55.151842 2181 webserver.cc:460] Webserver started at http://127.31.74.193:40659/ using document root <none> and password file <none>
I20260806 02:58:55.152994 2181 fs_manager.cc:359] Metadata directory not provided
I20260806 02:58:55.153170 2181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:58:55.153672 2181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:58:55.159126 2181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-0/data/instance:
uuid: "3b10eae1e6764a379aa2c9d6a4fec6f0"
format_stamp: "Formatted at 2026-08-06 02:58:55 on dist-test-slave-dhph"
server_key: "cb4463248707e89f12bbce8dd6bcdbb5"
server_key_iv: "4d46753b5ca468623247799b6fa2a241"
server_key_version: "encryptionkey@0"
I20260806 02:58:55.161129 2181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-0/wal/instance:
uuid: "3b10eae1e6764a379aa2c9d6a4fec6f0"
format_stamp: "Formatted at 2026-08-06 02:58:55 on dist-test-slave-dhph"
server_key: "cb4463248707e89f12bbce8dd6bcdbb5"
server_key_iv: "4d46753b5ca468623247799b6fa2a241"
server_key_version: "encryptionkey@0"
I20260806 02:58:55.169865 2181 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.009s sys 0.001s
I20260806 02:58:55.174988 2196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:58:55.178462 2181 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.004s sys 0.000s
I20260806 02:58:55.178776 2181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-0/wal
uuid: "3b10eae1e6764a379aa2c9d6a4fec6f0"
format_stamp: "Formatted at 2026-08-06 02:58:55 on dist-test-slave-dhph"
server_key: "cb4463248707e89f12bbce8dd6bcdbb5"
server_key_iv: "4d46753b5ca468623247799b6fa2a241"
server_key_version: "encryptionkey@0"
I20260806 02:58:55.179029 2181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:58:55.216176 2181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:58:55.221953 2181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:58:55.222250 2181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:58:55.223364 2181 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:58:55.224895 2181 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:58:55.224978 2181 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:58:55.225075 2181 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:58:55.225123 2181 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:58:55.269775 2181 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.193:45305
I20260806 02:58:55.269876 2310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.193:45305 every 8 connection(s)
I20260806 02:58:55.271193 2181 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
I20260806 02:58:55.278311 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 2181
I20260806 02:58:55.278654 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-0/wal/instance
I20260806 02:58:55.279340 32043 external_mini_cluster.cc:1428] Setting key e16e490ead2dc2b53891e4a7fc96f19f
Aug 06 02:58:55 dist-test-slave-dhph krb5kdc[2074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985135, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.193@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 02:58:55.298926 2313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:55.274246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.193:49783 (local address 127.31.74.254:43945)
0806 02:58:55.274812 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:55.274819 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:55.275799 (+ 980us) server_negotiation.cc:407] Connection header received
0806 02:58:55.277794 (+ 1995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:55.277800 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:55.277900 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:55.278022 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:55.279838 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:55.280814 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:55.284207 (+ 3393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:55.284811 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:55.284969 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:55.289740 (+ 4771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:58:55.289759 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:58:55.289767 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:58:55.289822 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:58:55.293985 (+ 4163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:55.295487 (+ 1502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:55.295494 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:55.295497 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:55.295747 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:55.296393 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:55.296397 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:55.296400 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:55.296803 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:58:55.296982 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:55.298153 (+ 1171us) server_negotiation.cc:299] Negotiation successful
0806 02:58:55.298403 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":255,"threads_started":1}
I20260806 02:58:55.300343 2314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:55.274800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:43945 (local address 127.31.74.193:49783)
0806 02:58:55.275549 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:55.275594 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 02:58:55.277474 (+ 1880us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:58:55.278287 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:58:55.278296 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:58:55.279015 (+ 719us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:58:55.279657 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:55.279677 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:55.281008 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:55.281012 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:55.283989 (+ 2977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:55.284015 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:55.285090 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:55.285101 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:55.285486 (+ 385us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:55.286381 (+ 895us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:58:55.286420 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:58:55.289446 (+ 3026us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:58:55.294216 (+ 4770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:58:55.294231 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:58:55.294255 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:58:55.295080 (+ 825us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:58:55.295920 (+ 840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:58:55.295924 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:58:55.295929 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:58:55.296233 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:58:55.296987 (+ 754us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:58:55.296998 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:58:55.297563 (+ 565us) client_negotiation.cc:769] Sending connection context
0806 02:58:55.297913 (+ 350us) client_negotiation.cc:240] Negotiation successful
0806 02:58:55.298554 (+ 641us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":307,"threads_started":1}
I20260806 02:58:55.302279 2311 heartbeater.cc:347] Connected to a master server at 127.31.74.254:43945
I20260806 02:58:55.302733 2311 heartbeater.cc:464] Registering TS with master...
I20260806 02:58:55.303522 2311 heartbeater.cc:511] Master 127.31.74.254:43945 requested a full tablet report, sending...
I20260806 02:58:55.306483 2120 ts_manager.cc:194] Registered new tserver with Master: 3b10eae1e6764a379aa2c9d6a4fec6f0 (127.31.74.193:45305)
WARNING: no policy specified for kudu/127.31.74.194@KRBTEST.COM; defaulting to no policy
I20260806 02:58:55.308411 2120 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.74.193@KRBTEST.COM'} at 127.31.74.193:49783
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.194.keytab.
Entry for principal kudu/127.31.74.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.194.keytab.
I20260806 02:58:55.349197 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.194:0
--local_ip_for_outbound_sockets=127.31.74.194
--webserver_interface=127.31.74.194
--webserver_port=0
--tserver_master_addrs=127.31.74.254:43945
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.194.keytab
--principal=kudu/127.31.74.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:43477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:58:55.745535 2317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:58:55.745886 2317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:58:55.746057 2317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:58:55.754895 2317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:58:55.755070 2317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:58:55.755263 2317 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.194
I20260806 02:58:55.767096 2317 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:43477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.194.keytab
--principal=kudu/127.31.74.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.74.194
--webserver_port=0
--tserver_master_addrs=127.31.74.254:43945
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.194
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:58:55.768749 2317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:58:55.770273 2317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:58:55.786496 2322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:58:55.786711 2323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:58:55.787539 2325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:58:55.792958 2317 server_base.cc:1034] running on GCE node
I20260806 02:58:55.793697 2317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:58:55.795074 2317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:58:55.796350 2317 hybrid_clock.cc:648] HybridClock initialized: now 1785985135796276 us; error 61 us; skew 500 ppm
I20260806 02:58:55.796855 2317 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:58:55 dist-test-slave-dhph krb5kdc[2074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985135, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:58:55.801275 2317 init.cc:377] Logged in from keytab as kudu/127.31.74.194@KRBTEST.COM (short username kudu)
I20260806 02:58:55.803253 2317 webserver.cc:460] Webserver started at http://127.31.74.194:38271/ using document root <none> and password file <none>
I20260806 02:58:55.804121 2317 fs_manager.cc:359] Metadata directory not provided
I20260806 02:58:55.804247 2317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:58:55.804605 2317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:58:55.808192 2317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-1/data/instance:
uuid: "7c9f14a8501d40a09843230f66fbd87c"
format_stamp: "Formatted at 2026-08-06 02:58:55 on dist-test-slave-dhph"
server_key: "2f90fbf47fbe5ea5d63e459d8385cdda"
server_key_iv: "9d360e2039d39fb94ff6f6539c991394"
server_key_version: "encryptionkey@0"
I20260806 02:58:55.809129 2317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-1/wal/instance:
uuid: "7c9f14a8501d40a09843230f66fbd87c"
format_stamp: "Formatted at 2026-08-06 02:58:55 on dist-test-slave-dhph"
server_key: "2f90fbf47fbe5ea5d63e459d8385cdda"
server_key_iv: "9d360e2039d39fb94ff6f6539c991394"
server_key_version: "encryptionkey@0"
I20260806 02:58:55.815653 2317 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260806 02:58:55.820490 2332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:58:55.822449 2317 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:58:55.822736 2317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-1/wal
uuid: "7c9f14a8501d40a09843230f66fbd87c"
format_stamp: "Formatted at 2026-08-06 02:58:55 on dist-test-slave-dhph"
server_key: "2f90fbf47fbe5ea5d63e459d8385cdda"
server_key_iv: "9d360e2039d39fb94ff6f6539c991394"
server_key_version: "encryptionkey@0"
I20260806 02:58:55.822975 2317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:58:55.854238 2317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:58:55.859993 2317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:58:55.860383 2317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:58:55.861559 2317 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:58:55.863916 2317 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:58:55.864022 2317 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:58:55.864200 2317 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:58:55.864286 2317 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:58:55.915764 2317 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.194:33111
I20260806 02:58:55.915979 2445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.194:33111 every 8 connection(s)
I20260806 02:58:55.917299 2317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
I20260806 02:58:55.919378 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 2317
I20260806 02:58:55.919629 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-1/wal/instance
I20260806 02:58:55.920133 32043 external_mini_cluster.cc:1428] Setting key 05bad1de5594748ffc146fb7a9afe7f0
Aug 06 02:58:55 dist-test-slave-dhph krb5kdc[2074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985135, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.194@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 02:58:55.948720 2448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:55.920588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.194:37949 (local address 127.31.74.254:43945)
0806 02:58:55.921115 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:55.921120 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:55.922609 (+ 1489us) server_negotiation.cc:407] Connection header received
0806 02:58:55.925826 (+ 3217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:55.925836 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:55.926179 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:55.926495 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:55.929060 (+ 2565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:55.929993 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:55.931985 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:55.932721 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:55.933026 (+ 305us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:55.939413 (+ 6387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:58:55.939420 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:58:55.939424 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:58:55.939469 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:58:55.943270 (+ 3801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:55.944670 (+ 1400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:55.944675 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:55.944710 (+ 35us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:55.944863 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:55.945690 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:55.945695 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:55.945698 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:55.946287 (+ 589us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:58:55.946448 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:55.948037 (+ 1589us) server_negotiation.cc:299] Negotiation successful
0806 02:58:55.948355 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":257,"threads_started":1}
WARNING: no policy specified for kudu/127.31.74.195@KRBTEST.COM; defaulting to no policy
I20260806 02:58:55.950212 2449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:55.921288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:43945 (local address 127.31.74.194:37949)
0806 02:58:55.922416 (+ 1128us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:55.922475 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0806 02:58:55.925175 (+ 2700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:58:55.926911 (+ 1736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:58:55.926923 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:58:55.927989 (+ 1066us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:58:55.928816 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:55.928844 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:55.930206 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:55.930213 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:55.931799 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:55.931816 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:55.933191 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:55.933205 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:55.933766 (+ 561us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:55.934850 (+ 1084us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:58:55.934891 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:58:55.938982 (+ 4091us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:58:55.943511 (+ 4529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:58:55.943526 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:58:55.943552 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:58:55.944459 (+ 907us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:58:55.945023 (+ 564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:58:55.945032 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:58:55.945039 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:58:55.945486 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:58:55.946493 (+ 1007us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:58:55.946507 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:58:55.947201 (+ 694us) client_negotiation.cc:769] Sending connection context
0806 02:58:55.947863 (+ 662us) client_negotiation.cc:240] Negotiation successful
0806 02:58:55.948598 (+ 735us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":764,"thread_start_us":431,"threads_started":1}
I20260806 02:58:55.952638 2446 heartbeater.cc:347] Connected to a master server at 127.31.74.254:43945
I20260806 02:58:55.953078 2446 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.195@KRBTEST.COM" created.
I20260806 02:58:55.954066 2446 heartbeater.cc:511] Master 127.31.74.254:43945 requested a full tablet report, sending...
I20260806 02:58:55.956310 2120 ts_manager.cc:194] Registered new tserver with Master: 7c9f14a8501d40a09843230f66fbd87c (127.31.74.194:33111)
I20260806 02:58:55.957455 2120 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.74.194@KRBTEST.COM'} at 127.31.74.194:37949
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.195.keytab.
Entry for principal kudu/127.31.74.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.195.keytab.
I20260806 02:58:55.987979 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.195:0
--local_ip_for_outbound_sockets=127.31.74.195
--webserver_interface=127.31.74.195
--webserver_port=0
--tserver_master_addrs=127.31.74.254:43945
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.195.keytab
--principal=kudu/127.31.74.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:43477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:58:56.312991 2311 heartbeater.cc:503] Master 127.31.74.254:43945 was elected leader, sending a full tablet report...
W20260806 02:58:56.376293 2452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:58:56.376847 2452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:58:56.376974 2452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:58:56.387357 2452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:58:56.387508 2452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:58:56.387647 2452 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.195
I20260806 02:58:56.399609 2452 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:43477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.195.keytab
--principal=kudu/127.31.74.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.74.195
--webserver_port=0
--tserver_master_addrs=127.31.74.254:43945
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.195
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:58:56.401986 2452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:58:56.403596 2452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:58:56.420984 2460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:58:56.421125 2457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:58:56.420966 2458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:58:56.424757 2452 server_base.cc:1034] running on GCE node
I20260806 02:58:56.425596 2452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:58:56.426934 2452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:58:56.428608 2452 hybrid_clock.cc:648] HybridClock initialized: now 1785985136428529 us; error 56 us; skew 500 ppm
I20260806 02:58:56.429020 2452 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:58:56 dist-test-slave-dhph krb5kdc[2074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985136, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:58:56.432729 2452 init.cc:377] Logged in from keytab as kudu/127.31.74.195@KRBTEST.COM (short username kudu)
I20260806 02:58:56.434474 2452 webserver.cc:460] Webserver started at http://127.31.74.195:43791/ using document root <none> and password file <none>
I20260806 02:58:56.435251 2452 fs_manager.cc:359] Metadata directory not provided
I20260806 02:58:56.435367 2452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:58:56.435638 2452 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:58:56.438866 2452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-2/data/instance:
uuid: "5f3f320061444c55af21b12ed538ee7b"
format_stamp: "Formatted at 2026-08-06 02:58:56 on dist-test-slave-dhph"
server_key: "6af933b9b30825c645dcf5d5315fdc64"
server_key_iv: "295e96db273122011aa03ae85e4999a5"
server_key_version: "encryptionkey@0"
I20260806 02:58:56.439707 2452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-2/wal/instance:
uuid: "5f3f320061444c55af21b12ed538ee7b"
format_stamp: "Formatted at 2026-08-06 02:58:56 on dist-test-slave-dhph"
server_key: "6af933b9b30825c645dcf5d5315fdc64"
server_key_iv: "295e96db273122011aa03ae85e4999a5"
server_key_version: "encryptionkey@0"
I20260806 02:58:56.445926 2452 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260806 02:58:56.452041 2467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:58:56.453884 2452 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:58:56.454053 2452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-2/wal
uuid: "5f3f320061444c55af21b12ed538ee7b"
format_stamp: "Formatted at 2026-08-06 02:58:56 on dist-test-slave-dhph"
server_key: "6af933b9b30825c645dcf5d5315fdc64"
server_key_iv: "295e96db273122011aa03ae85e4999a5"
server_key_version: "encryptionkey@0"
I20260806 02:58:56.454348 2452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:58:56.499142 2452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:58:56.505501 2452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:58:56.505793 2452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:58:56.506958 2452 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:58:56.508651 2452 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:58:56.508767 2452 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:58:56.508919 2452 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:58:56.508986 2452 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:58:56.553218 2452 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.195:35699
I20260806 02:58:56.553341 2583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.195:35699 every 8 connection(s)
I20260806 02:58:56.554970 2452 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
I20260806 02:58:56.556854 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 2452
I20260806 02:58:56.557104 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-2/wal/instance
I20260806 02:58:56.557588 32043 external_mini_cluster.cc:1428] Setting key 40d3199399220fec6ff6dfff1b75f64e
Aug 06 02:58:56 dist-test-slave-dhph krb5kdc[2074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985136, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.195@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 02:58:56.581657 2586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:56.558005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.195:47355 (local address 127.31.74.254:43945)
0806 02:58:56.559031 (+ 1026us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:56.559037 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:56.559929 (+ 892us) server_negotiation.cc:407] Connection header received
0806 02:58:56.562347 (+ 2418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:56.562351 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:56.562436 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:56.562536 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:56.564345 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:56.565227 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:56.567116 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:56.567972 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:56.568097 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:56.573223 (+ 5126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:58:56.573229 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:58:56.573234 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:58:56.573281 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:58:56.577043 (+ 3762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:56.578225 (+ 1182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:56.578230 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:56.578233 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:56.578381 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:56.579333 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:56.579339 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:56.579342 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:56.579690 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:58:56.579913 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:56.581132 (+ 1219us) server_negotiation.cc:299] Negotiation successful
0806 02:58:56.581351 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":885,"thread_start_us":713,"threads_started":1}
I20260806 02:58:56.582866 2587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:56.558813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:43945 (local address 127.31.74.195:47355)
0806 02:58:56.559730 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:56.559796 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0806 02:58:56.562027 (+ 2231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:58:56.562894 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:58:56.562904 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:58:56.563499 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:58:56.564117 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:56.564136 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:56.565411 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:56.565416 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:56.566854 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:56.566868 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:56.568356 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:56.568379 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:56.568606 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:56.569501 (+ 895us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:58:56.569528 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:58:56.572980 (+ 3452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:58:56.577260 (+ 4280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:58:56.577270 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:58:56.577286 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:58:56.578012 (+ 726us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:58:56.578531 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:58:56.578535 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:58:56.578538 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:58:56.579183 (+ 645us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:58:56.579924 (+ 741us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:58:56.579931 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:58:56.580616 (+ 685us) client_negotiation.cc:769] Sending connection context
0806 02:58:56.580982 (+ 366us) client_negotiation.cc:240] Negotiation successful
0806 02:58:56.581465 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":374,"threads_started":1}
I20260806 02:58:56.584636 2584 heartbeater.cc:347] Connected to a master server at 127.31.74.254:43945
I20260806 02:58:56.585000 2584 heartbeater.cc:464] Registering TS with master...
I20260806 02:58:56.586117 2584 heartbeater.cc:511] Master 127.31.74.254:43945 requested a full tablet report, sending...
I20260806 02:58:56.588235 2120 ts_manager.cc:194] Registered new tserver with Master: 5f3f320061444c55af21b12ed538ee7b (127.31.74.195:35699)
I20260806 02:58:56.589231 2120 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.74.195@KRBTEST.COM'} at 127.31.74.195:47355
I20260806 02:58:56.603148 32043 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260806 02:58:56.654251 2586 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:58:56.650359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38534 (local address 127.31.74.254:43945)
0806 02:58:56.650650 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:56.650656 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:56.651572 (+ 916us) server_negotiation.cc:407] Connection header received
0806 02:58:56.651902 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:56.651907 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:56.651987 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:56.652084 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:56.653817 (+ 1733us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38534: BlockingRecv error: recv got EOF from 127.0.0.1:38534 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
I20260806 02:58:56.660663 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 2181
W20260806 02:58:56.664690 32043 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 2181: Not authorized: Client connection negotiation failed: client connection to 127.31.74.193:45305: server requires authentication, but client does not have Kerberos credentials available
I20260806 02:58:56.664990 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 2181
W20260806 02:58:56.665148 2596 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:58:56.662154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42700 (local address 127.31.74.193:45305)
0806 02:58:56.663024 (+ 870us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:56.663032 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:56.663062 (+ 30us) server_negotiation.cc:407] Connection header received
0806 02:58:56.663144 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:56.663152 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:56.663399 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:56.663614 (+ 215us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:56.664631 (+ 1017us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42700: BlockingRecv error: recv got EOF from 127.0.0.1:42700 (error 108)
Metrics: {"server-negotiator.queue_time_us":670,"thread_start_us":341,"threads_started":1}
I20260806 02:58:56.686318 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 2317
W20260806 02:58:56.690008 32043 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 2317: Not authorized: Client connection negotiation failed: client connection to 127.31.74.194:33111: server requires authentication, but client does not have Kerberos credentials available
I20260806 02:58:56.690297 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 2317
I20260806 02:58:56.712167 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 2452
W20260806 02:58:56.716146 32043 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 2452: Not authorized: Client connection negotiation failed: client connection to 127.31.74.195:35699: server requires authentication, but client does not have Kerberos credentials available
I20260806 02:58:56.716365 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 2452
I20260806 02:58:56.737411 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 2087
I20260806 02:58:56.739085 2150 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:58:56.859714 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 2087
2026-08-06T02:58:56Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (4789 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785985105255475-32043-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:58:56 dist-test-slave-dhph krb5kdc[2602](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:58:56 dist-test-slave-dhph krb5kdc[2602](info): set up 2 sockets
Aug 06 02:58:56 dist-test-slave-dhph krb5kdc[2602](info): commencing operation
krb5kdc: starting...
W20260806 02:58:58.081835 32043 mini_kdc.cc:121] Time spent starting KDC: real 1.172s user 0.004s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785985105255475-32043-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785985105255475-32043-0/krb5kdc/test-user.keytab.
Aug 06 02:58:58 dist-test-slave-dhph krb5kdc[2602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985138, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:58:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:58:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.74.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab.
Entry for principal kudu/127.31.74.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab.
I20260806 02:58:58.368435 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785985105255475-32043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785985105255475-32043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785985105255475-32043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:36971
--webserver_interface=127.31.74.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab
--principal=kudu/127.31.74.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:36405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.74.254:36971
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:58:58.799683 2615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:58:58.800079 2615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:58:58.800254 2615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:58:58.808677 2615 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:58:58.808952 2615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:58:58.809194 2615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:58:58.809381 2615 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:58:58.809496 2615 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:58:58.822551 2615 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:36405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785985105255475-32043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785985105255475-32043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.74.254:36971
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab
--principal=kudu/127.31.74.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:36971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.74.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785985105255475-32043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:58:58.824003 2615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:58:58.825390 2615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:58:58.838836 2621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:58:58.838817 2620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:58:58.839089 2623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:58:58.840399 2615 server_base.cc:1034] running on GCE node
I20260806 02:58:58.841998 2615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:58:58.844146 2615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:58:58.845489 2615 hybrid_clock.cc:648] HybridClock initialized: now 1785985138845432 us; error 39 us; skew 500 ppm
I20260806 02:58:58.845961 2615 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:58:58 dist-test-slave-dhph krb5kdc[2602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985138, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:58:58.850504 2615 init.cc:377] Logged in from keytab as kudu/127.31.74.254@KRBTEST.COM (short username kudu)
I20260806 02:58:58.852795 2615 webserver.cc:460] Webserver started at http://127.31.74.254:43221/ using document root <none> and password file <none>
I20260806 02:58:58.853611 2615 fs_manager.cc:359] Metadata directory not provided
I20260806 02:58:58.853740 2615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:58:58.854032 2615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:58:58.858436 2615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785985105255475-32043-0/minicluster-data/master-0/data/instance:
uuid: "def411200b64492da12264a897f1ed29"
format_stamp: "Formatted at 2026-08-06 02:58:58 on dist-test-slave-dhph"
server_key: "9c390b9182b6d42d4f1aa36981edd061"
server_key_iv: "520ffe349af09585fcd63ae61a0ddf9c"
server_key_version: "encryptionkey@0"
I20260806 02:58:58.860512 2615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785985105255475-32043-0/minicluster-data/master-0/wal/instance:
uuid: "def411200b64492da12264a897f1ed29"
format_stamp: "Formatted at 2026-08-06 02:58:58 on dist-test-slave-dhph"
server_key: "9c390b9182b6d42d4f1aa36981edd061"
server_key_iv: "520ffe349af09585fcd63ae61a0ddf9c"
server_key_version: "encryptionkey@0"
I20260806 02:58:58.870033 2615 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.009s sys 0.000s
I20260806 02:58:58.875463 2630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:58:58.877283 2615 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 02:58:58.877519 2615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785985105255475-32043-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785985105255475-32043-0/minicluster-data/master-0/wal
uuid: "def411200b64492da12264a897f1ed29"
format_stamp: "Formatted at 2026-08-06 02:58:58 on dist-test-slave-dhph"
server_key: "9c390b9182b6d42d4f1aa36981edd061"
server_key_iv: "520ffe349af09585fcd63ae61a0ddf9c"
server_key_version: "encryptionkey@0"
I20260806 02:58:58.877997 2615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785985105255475-32043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785985105255475-32043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785985105255475-32043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:58:58.896673 2615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:58:58.905035 2615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:58:58.905541 2615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:58:58.935384 2615 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.254:36971
I20260806 02:58:58.935402 2682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.254:36971 every 8 connection(s)
I20260806 02:58:58.937084 2615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
I20260806 02:58:58.942118 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 2615
I20260806 02:58:58.942407 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785985105255475-32043-0/minicluster-data/master-0/wal/instance
I20260806 02:58:58.943523 32043 external_mini_cluster.cc:1428] Setting key b61321bba89cfe0765308943abc7fa4b
I20260806 02:58:58.944566 2683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:58:58.963518 2683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P def411200b64492da12264a897f1ed29: Bootstrap starting.
Aug 06 02:58:58 dist-test-slave-dhph krb5kdc[2602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985138, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 02:58:58.968425 2683 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P def411200b64492da12264a897f1ed29: Neither blocks nor log segments found. Creating new log.
I20260806 02:58:58.970438 2683 log.cc:824] T 00000000000000000000000000000000 P def411200b64492da12264a897f1ed29: Log is configured to *not* fsync() on all Append() calls
I20260806 02:58:58.974660 2683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P def411200b64492da12264a897f1ed29: No bootstrap required, opened a new log
I20260806 02:58:58.979164 2686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:58.946006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35972 (local address 127.31.74.254:36971)
0806 02:58:58.947449 (+ 1443us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:58.947463 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:58.947508 (+ 45us) server_negotiation.cc:407] Connection header received
0806 02:58:58.950031 (+ 2523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:58.950059 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:58.950575 (+ 516us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:58.951064 (+ 489us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:58.953200 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:58.954986 (+ 1786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:58.958134 (+ 3148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:58.960781 (+ 2647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:58.960965 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:58.967473 (+ 6508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:58:58.967483 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:58:58.967500 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:58:58.967547 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:58:58.972745 (+ 5198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:58.973747 (+ 1002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:58.973756 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:58.973766 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:58.974010 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:58.974515 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:58.974521 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:58.974524 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:58.975440 (+ 916us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:58:58.975827 (+ 387us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:58.976589 (+ 762us) server_negotiation.cc:299] Negotiation successful
0806 02:58:58.977556 (+ 967us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1130,"thread_start_us":671,"threads_started":1}
I20260806 02:58:58.984319 32043 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:58:58.985690 2683 raft_consensus.cc:348] T 00000000000000000000000000000000 P def411200b64492da12264a897f1ed29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "def411200b64492da12264a897f1ed29" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36971 } }
I20260806 02:58:58.986230 2683 raft_consensus.cc:374] T 00000000000000000000000000000000 P def411200b64492da12264a897f1ed29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:58:58.986528 2683 raft_consensus.cc:729] T 00000000000000000000000000000000 P def411200b64492da12264a897f1ed29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: def411200b64492da12264a897f1ed29, State: Initialized, Role: FOLLOWER
I20260806 02:58:58.987432 2683 consensus_queue.cc:260] T 00000000000000000000000000000000 P def411200b64492da12264a897f1ed29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "def411200b64492da12264a897f1ed29" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36971 } }
I20260806 02:58:58.987802 2683 raft_consensus.cc:388] T 00000000000000000000000000000000 P def411200b64492da12264a897f1ed29 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:58:58.987977 2683 raft_consensus.cc:482] T 00000000000000000000000000000000 P def411200b64492da12264a897f1ed29 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:58:58.988181 2683 raft_consensus.cc:3037] T 00000000000000000000000000000000 P def411200b64492da12264a897f1ed29 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:58:58.992137 2683 raft_consensus.cc:504] T 00000000000000000000000000000000 P def411200b64492da12264a897f1ed29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "def411200b64492da12264a897f1ed29" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36971 } }
I20260806 02:58:58.993242 2683 leader_election.cc:302] T 00000000000000000000000000000000 P def411200b64492da12264a897f1ed29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: def411200b64492da12264a897f1ed29; no voters:
I20260806 02:58:58.994069 2683 leader_election.cc:288] T 00000000000000000000000000000000 P def411200b64492da12264a897f1ed29 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:58:58.994151 2688 raft_consensus.cc:2781] T 00000000000000000000000000000000 P def411200b64492da12264a897f1ed29 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:58:58.995930 2688 raft_consensus.cc:686] T 00000000000000000000000000000000 P def411200b64492da12264a897f1ed29 [term 1 LEADER]: Becoming Leader. State: Replica: def411200b64492da12264a897f1ed29, State: Running, Role: LEADER
I20260806 02:58:58.996963 2683 sys_catalog.cc:564] T 00000000000000000000000000000000 P def411200b64492da12264a897f1ed29 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:58:58.996672 2688 consensus_queue.cc:237] T 00000000000000000000000000000000 P def411200b64492da12264a897f1ed29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "def411200b64492da12264a897f1ed29" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36971 } }
I20260806 02:58:59.001545 2690 sys_catalog.cc:455] T 00000000000000000000000000000000 P def411200b64492da12264a897f1ed29 [sys.catalog]: SysCatalogTable state changed. Reason: New leader def411200b64492da12264a897f1ed29. Latest consensus state: current_term: 1 leader_uuid: "def411200b64492da12264a897f1ed29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "def411200b64492da12264a897f1ed29" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36971 } } }
I20260806 02:58:59.001533 2689 sys_catalog.cc:455] T 00000000000000000000000000000000 P def411200b64492da12264a897f1ed29 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "def411200b64492da12264a897f1ed29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "def411200b64492da12264a897f1ed29" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36971 } } }
I20260806 02:58:59.002051 2690 sys_catalog.cc:458] T 00000000000000000000000000000000 P def411200b64492da12264a897f1ed29 [sys.catalog]: This master's current role is: LEADER
I20260806 02:58:59.002127 2689 sys_catalog.cc:458] T 00000000000000000000000000000000 P def411200b64492da12264a897f1ed29 [sys.catalog]: This master's current role is: LEADER
I20260806 02:58:59.013557 2698 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:58:59.019773 2698 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:58:59.026561 2706 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 02:58:59.033507 2698 catalog_manager.cc:1347] Generated new cluster ID: 70493d54086e4a5ba6a8d02aa64b076c
I20260806 02:58:59.033706 2698 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:58:59.047053 2698 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:58:59.048419 2698 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:58:59.059222 2686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:59.044530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35986 (local address 127.31.74.254:36971)
0806 02:58:59.044718 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:59.044729 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:59.046094 (+ 1365us) server_negotiation.cc:407] Connection header received
0806 02:58:59.046311 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:59.046316 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:59.046407 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:59.046526 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:59.050005 (+ 3479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:59.051348 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:59.053772 (+ 2424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:59.054664 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:59.054813 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:59.056424 (+ 1611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:58:59.056432 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:58:59.056436 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:58:59.057201 (+ 765us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:58:59.057272 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:58:59.057297 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:58:59.057448 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:59.058589 (+ 1141us) server_negotiation.cc:299] Negotiation successful
0806 02:58:59.058825 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 02:58:59.060861 2698 catalog_manager.cc:5868] T 00000000000000000000000000000000 P def411200b64492da12264a897f1ed29: Generated new TSK 0
I20260806 02:58:59.063205 2698 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:58:59.072739 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 2615
I20260806 02:58:59.075301 2678 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:58:59.181567 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 2615
2026-08-06T02:58:59Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2319 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:58:59 dist-test-slave-dhph krb5kdc[2717](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:58:59 dist-test-slave-dhph krb5kdc[2717](info): set up 2 sockets
Aug 06 02:58:59 dist-test-slave-dhph krb5kdc[2717](info): commencing operation
krb5kdc: starting...
W20260806 02:59:00.458115 32043 mini_kdc.cc:121] Time spent starting KDC: real 1.227s user 0.003s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/krb5kdc/test-user.keytab.
Aug 06 02:59:00 dist-test-slave-dhph krb5kdc[2717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985140, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:59:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:59:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.74.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab.
Entry for principal kudu/127.31.74.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab.
I20260806 02:59:00.779855 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:38741
--webserver_interface=127.31.74.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab
--principal=kudu/127.31.74.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:36733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.74.254:38741
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:59:01.185423 2730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:59:01.186035 2730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:59:01.186156 2730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:59:01.196920 2730 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:59:01.197078 2730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:59:01.197130 2730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:59:01.197183 2730 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:59:01.197237 2730 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:59:01.210033 2730 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:36733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.74.254:38741
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab
--principal=kudu/127.31.74.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:38741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.74.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:59:01.211570 2730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:59:01.213330 2730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:59:01.228745 2736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:01.228718 2738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:01.229667 2735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:59:01.229876 2730 server_base.cc:1034] running on GCE node
I20260806 02:59:01.231030 2730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:59:01.232928 2730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:59:01.234431 2730 hybrid_clock.cc:648] HybridClock initialized: now 1785985141234264 us; error 59 us; skew 500 ppm
I20260806 02:59:01.235198 2730 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:59:01 dist-test-slave-dhph krb5kdc[2717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985141, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:59:01.244607 2730 init.cc:377] Logged in from keytab as kudu/127.31.74.254@KRBTEST.COM (short username kudu)
I20260806 02:59:01.246654 2730 webserver.cc:460] Webserver started at http://127.31.74.254:40035/ using document root <none> and password file <none>
I20260806 02:59:01.247452 2730 fs_manager.cc:359] Metadata directory not provided
I20260806 02:59:01.247594 2730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:59:01.247918 2730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:59:01.251736 2730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/minicluster-data/master-0/data/instance:
uuid: "4e9000f4a03249bba70487e7052f9d3b"
format_stamp: "Formatted at 2026-08-06 02:59:01 on dist-test-slave-dhph"
server_key: "a332ee3cc36c854a21fb8318693ab8ff"
server_key_iv: "eb2f7c29db3a6f6265fecda967d26135"
server_key_version: "encryptionkey@0"
I20260806 02:59:01.252630 2730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/minicluster-data/master-0/wal/instance:
uuid: "4e9000f4a03249bba70487e7052f9d3b"
format_stamp: "Formatted at 2026-08-06 02:59:01 on dist-test-slave-dhph"
server_key: "a332ee3cc36c854a21fb8318693ab8ff"
server_key_iv: "eb2f7c29db3a6f6265fecda967d26135"
server_key_version: "encryptionkey@0"
I20260806 02:59:01.260520 2730 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20260806 02:59:01.264784 2745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:01.266551 2730 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:59:01.266852 2730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/minicluster-data/master-0/wal
uuid: "4e9000f4a03249bba70487e7052f9d3b"
format_stamp: "Formatted at 2026-08-06 02:59:01 on dist-test-slave-dhph"
server_key: "a332ee3cc36c854a21fb8318693ab8ff"
server_key_iv: "eb2f7c29db3a6f6265fecda967d26135"
server_key_version: "encryptionkey@0"
I20260806 02:59:01.267073 2730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:59:01.280095 2730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:59:01.286296 2730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:59:01.286655 2730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:59:01.309317 2730 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.254:38741
I20260806 02:59:01.309365 2797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.254:38741 every 8 connection(s)
I20260806 02:59:01.311059 2730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
I20260806 02:59:01.315392 2798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:59:01.321336 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 2730
I20260806 02:59:01.321563 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/minicluster-data/master-0/wal/instance
I20260806 02:59:01.322561 32043 external_mini_cluster.cc:1428] Setting key 8918c416e946af600bd1a932431092d5
I20260806 02:59:01.328806 2798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e9000f4a03249bba70487e7052f9d3b: Bootstrap starting.
I20260806 02:59:01.332252 2798 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e9000f4a03249bba70487e7052f9d3b: Neither blocks nor log segments found. Creating new log.
I20260806 02:59:01.333429 2798 log.cc:824] T 00000000000000000000000000000000 P 4e9000f4a03249bba70487e7052f9d3b: Log is configured to *not* fsync() on all Append() calls
I20260806 02:59:01.337697 2798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e9000f4a03249bba70487e7052f9d3b: No bootstrap required, opened a new log
Aug 06 02:59:01 dist-test-slave-dhph krb5kdc[2717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985140, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 02:59:01.348979 2798 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e9000f4a03249bba70487e7052f9d3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e9000f4a03249bba70487e7052f9d3b" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 38741 } }
I20260806 02:59:01.349493 2798 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e9000f4a03249bba70487e7052f9d3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:59:01.349637 2798 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e9000f4a03249bba70487e7052f9d3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e9000f4a03249bba70487e7052f9d3b, State: Initialized, Role: FOLLOWER
I20260806 02:59:01.350441 2798 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e9000f4a03249bba70487e7052f9d3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e9000f4a03249bba70487e7052f9d3b" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 38741 } }
I20260806 02:59:01.350814 2798 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e9000f4a03249bba70487e7052f9d3b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:59:01.351002 2798 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e9000f4a03249bba70487e7052f9d3b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:59:01.351210 2798 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e9000f4a03249bba70487e7052f9d3b [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:59:01.354535 2798 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e9000f4a03249bba70487e7052f9d3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e9000f4a03249bba70487e7052f9d3b" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 38741 } }
I20260806 02:59:01.355517 2798 leader_election.cc:302] T 00000000000000000000000000000000 P 4e9000f4a03249bba70487e7052f9d3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e9000f4a03249bba70487e7052f9d3b; no voters:
I20260806 02:59:01.355804 2801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:01.325235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60516 (local address 127.31.74.254:38741)
0806 02:59:01.326474 (+ 1239us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:01.326491 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:01.326538 (+ 47us) server_negotiation.cc:407] Connection header received
0806 02:59:01.329141 (+ 2603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:01.329168 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:01.329617 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:01.330059 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:01.331822 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:01.333362 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:01.335385 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:01.336504 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:01.336736 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:01.344320 (+ 7584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:01.344332 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:01.344359 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:01.344416 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:01.349505 (+ 5089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:01.350776 (+ 1271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:01.350787 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:01.350806 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:01.351173 (+ 367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:01.351761 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:01.351769 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:01.351771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:01.352771 (+ 1000us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:01.353073 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:01.353820 (+ 747us) server_negotiation.cc:299] Negotiation successful
0806 02:59:01.354387 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":860,"thread_start_us":596,"threads_started":1}
I20260806 02:59:01.356343 2798 leader_election.cc:288] T 00000000000000000000000000000000 P 4e9000f4a03249bba70487e7052f9d3b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:59:01.356521 2803 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e9000f4a03249bba70487e7052f9d3b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:59:01.359094 2803 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e9000f4a03249bba70487e7052f9d3b [term 1 LEADER]: Becoming Leader. State: Replica: 4e9000f4a03249bba70487e7052f9d3b, State: Running, Role: LEADER
I20260806 02:59:01.359822 2803 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e9000f4a03249bba70487e7052f9d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e9000f4a03249bba70487e7052f9d3b" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 38741 } }
I20260806 02:59:01.360298 2798 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e9000f4a03249bba70487e7052f9d3b [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:59:01.363438 2805 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e9000f4a03249bba70487e7052f9d3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e9000f4a03249bba70487e7052f9d3b. Latest consensus state: current_term: 1 leader_uuid: "4e9000f4a03249bba70487e7052f9d3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e9000f4a03249bba70487e7052f9d3b" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 38741 } } }
I20260806 02:59:01.364032 2805 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e9000f4a03249bba70487e7052f9d3b [sys.catalog]: This master's current role is: LEADER
I20260806 02:59:01.364758 2804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e9000f4a03249bba70487e7052f9d3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4e9000f4a03249bba70487e7052f9d3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e9000f4a03249bba70487e7052f9d3b" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 38741 } } }
I20260806 02:59:01.365474 2804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e9000f4a03249bba70487e7052f9d3b [sys.catalog]: This master's current role is: LEADER
I20260806 02:59:01.368199 2812 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:59:01.373736 2812 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:59:01.385710 2812 catalog_manager.cc:1347] Generated new cluster ID: cc5d6d699fa344f7a486369515e4e431
I20260806 02:59:01.386077 2812 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:59:01.406328 2812 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:59:01.407831 2812 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:59:01.416522 2812 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4e9000f4a03249bba70487e7052f9d3b: Generated new TSK 0
I20260806 02:59:01.417623 2812 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.74.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.193.keytab.
Entry for principal kudu/127.31.74.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.193.keytab.
I20260806 02:59:01.504619 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.193:0
--local_ip_for_outbound_sockets=127.31.74.193
--webserver_interface=127.31.74.193
--webserver_port=0
--tserver_master_addrs=127.31.74.254:38741
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.193.keytab
--principal=kudu/127.31.74.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:36733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:59:01.977317 2824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:59:01.977694 2824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:59:01.977814 2824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:59:01.988040 2824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:59:01.988195 2824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:59:01.988322 2824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.193
I20260806 02:59:02.000516 2824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:36733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.193.keytab
--principal=kudu/127.31.74.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.74.193
--webserver_port=0
--tserver_master_addrs=127.31.74.254:38741
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.193
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:59:02.002445 2824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:59:02.004252 2824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:59:02.023403 2832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:02.023283 2829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:02.023517 2830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:59:02.024385 2824 server_base.cc:1034] running on GCE node
I20260806 02:59:02.025149 2824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:59:02.027318 2824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:59:02.030874 2824 hybrid_clock.cc:648] HybridClock initialized: now 1785985142030751 us; error 87 us; skew 500 ppm
I20260806 02:59:02.031881 2824 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:59:02 dist-test-slave-dhph krb5kdc[2717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985142, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:59:02.043052 2824 init.cc:377] Logged in from keytab as kudu/127.31.74.193@KRBTEST.COM (short username kudu)
I20260806 02:59:02.046039 2824 webserver.cc:460] Webserver started at http://127.31.74.193:43077/ using document root <none> and password file <none>
I20260806 02:59:02.047081 2824 fs_manager.cc:359] Metadata directory not provided
I20260806 02:59:02.047276 2824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:59:02.047655 2824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:59:02.052445 2824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-0/data/instance:
uuid: "b5a23f82fbfc461a8cf24fe34758af1b"
format_stamp: "Formatted at 2026-08-06 02:59:02 on dist-test-slave-dhph"
server_key: "e6685e2075fd99650e3808edb158e4ae"
server_key_iv: "f0c1a4adda958b697e9f8ce41bc554b8"
server_key_version: "encryptionkey@0"
I20260806 02:59:02.053862 2824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-0/wal/instance:
uuid: "b5a23f82fbfc461a8cf24fe34758af1b"
format_stamp: "Formatted at 2026-08-06 02:59:02 on dist-test-slave-dhph"
server_key: "e6685e2075fd99650e3808edb158e4ae"
server_key_iv: "f0c1a4adda958b697e9f8ce41bc554b8"
server_key_version: "encryptionkey@0"
I20260806 02:59:02.060616 2824 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.004s
I20260806 02:59:02.065193 2839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:02.067317 2824 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260806 02:59:02.067587 2824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-0/wal
uuid: "b5a23f82fbfc461a8cf24fe34758af1b"
format_stamp: "Formatted at 2026-08-06 02:59:02 on dist-test-slave-dhph"
server_key: "e6685e2075fd99650e3808edb158e4ae"
server_key_iv: "f0c1a4adda958b697e9f8ce41bc554b8"
server_key_version: "encryptionkey@0"
I20260806 02:59:02.067859 2824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:59:02.103582 2824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:59:02.112323 2824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:59:02.112730 2824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:59:02.114038 2824 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:59:02.115860 2824 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:59:02.115991 2824 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:02.116173 2824 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:59:02.116269 2824 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:02.169660 2824 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.193:36901
I20260806 02:59:02.169822 2952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.193:36901 every 8 connection(s)
I20260806 02:59:02.172427 2824 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
I20260806 02:59:02.182165 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 2824
I20260806 02:59:02.182457 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-0/wal/instance
I20260806 02:59:02.183645 32043 external_mini_cluster.cc:1428] Setting key cc42740a5fd7b34f241222c79b72ce84
Aug 06 02:59:02 dist-test-slave-dhph krb5kdc[2717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985142, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.193@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 02:59:02.208264 2956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:02.177973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.193:38725 (local address 127.31.74.254:38741)
0806 02:59:02.178491 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:02.178499 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:02.178792 (+ 293us) server_negotiation.cc:407] Connection header received
0806 02:59:02.181311 (+ 2519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:02.181317 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:02.181409 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:02.181568 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:02.184789 (+ 3221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:02.187240 (+ 2451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:02.191558 (+ 4318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:02.192512 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:02.192848 (+ 336us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:02.198976 (+ 6128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:02.198984 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:02.198990 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:02.199071 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:02.203705 (+ 4634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:02.204772 (+ 1067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:02.204779 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:02.204783 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:02.204950 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:02.205804 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:02.205810 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:02.205813 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:02.206203 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:02.206417 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:02.207513 (+ 1096us) server_negotiation.cc:299] Negotiation successful
0806 02:59:02.207818 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":1196,"threads_started":1}
I20260806 02:59:02.209605 2955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:02.177489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:38741 (local address 127.31.74.193:38725)
0806 02:59:02.178545 (+ 1056us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:02.178638 (+ 93us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:02.180743 (+ 2105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:02.181989 (+ 1246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:02.182006 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:02.183290 (+ 1284us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:59:02.184460 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:02.184495 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:02.187507 (+ 3012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:02.187516 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:02.191299 (+ 3783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:02.191331 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:02.192897 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:02.192909 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:02.193322 (+ 413us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:02.194862 (+ 1540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:59:02.194910 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:59:02.198521 (+ 3611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:59:02.203952 (+ 5431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:02.203963 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:02.203994 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:02.204605 (+ 611us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:02.205085 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:02.205090 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:02.205095 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:02.205344 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:02.206385 (+ 1041us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:59:02.206395 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:59:02.206989 (+ 594us) client_negotiation.cc:769] Sending connection context
0806 02:59:02.207394 (+ 405us) client_negotiation.cc:240] Negotiation successful
0806 02:59:02.208065 (+ 671us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"spinlock_wait_cycles":22400,"thread_start_us":569,"threads_started":1}
I20260806 02:59:02.213811 2953 heartbeater.cc:347] Connected to a master server at 127.31.74.254:38741
I20260806 02:59:02.214243 2953 heartbeater.cc:464] Registering TS with master...
I20260806 02:59:02.215471 2953 heartbeater.cc:511] Master 127.31.74.254:38741 requested a full tablet report, sending...
I20260806 02:59:02.218915 2762 ts_manager.cc:194] Registered new tserver with Master: b5a23f82fbfc461a8cf24fe34758af1b (127.31.74.193:36901)
I20260806 02:59:02.221421 2762 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.74.193@KRBTEST.COM'} at 127.31.74.193:38725
WARNING: no policy specified for kudu/127.31.74.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.194.keytab.
Entry for principal kudu/127.31.74.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.194.keytab.
I20260806 02:59:02.273869 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.194:0
--local_ip_for_outbound_sockets=127.31.74.194
--webserver_interface=127.31.74.194
--webserver_port=0
--tserver_master_addrs=127.31.74.254:38741
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.194.keytab
--principal=kudu/127.31.74.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:36733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:59:02.714390 2959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:59:02.714745 2959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:59:02.714890 2959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:59:02.724424 2959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:59:02.724658 2959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:59:02.724818 2959 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.194
I20260806 02:59:02.740311 2959 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:36733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.194.keytab
--principal=kudu/127.31.74.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.74.194
--webserver_port=0
--tserver_master_addrs=127.31.74.254:38741
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.194
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:59:02.741844 2959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:59:02.743623 2959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:59:02.760895 2964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:02.760998 2965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:02.761242 2967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:59:02.762785 2959 server_base.cc:1034] running on GCE node
I20260806 02:59:02.763588 2959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:59:02.764948 2959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:59:02.766274 2959 hybrid_clock.cc:648] HybridClock initialized: now 1785985142766229 us; error 70 us; skew 500 ppm
I20260806 02:59:02.766750 2959 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:59:02 dist-test-slave-dhph krb5kdc[2717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985142, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:59:02.770905 2959 init.cc:377] Logged in from keytab as kudu/127.31.74.194@KRBTEST.COM (short username kudu)
I20260806 02:59:02.773475 2959 webserver.cc:460] Webserver started at http://127.31.74.194:35695/ using document root <none> and password file <none>
I20260806 02:59:02.774325 2959 fs_manager.cc:359] Metadata directory not provided
I20260806 02:59:02.774513 2959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:59:02.774945 2959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:59:02.778292 2959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-1/data/instance:
uuid: "cb1071102e414a35bd83ae06489d0392"
format_stamp: "Formatted at 2026-08-06 02:59:02 on dist-test-slave-dhph"
server_key: "0dab08749a3c813ce41dc8822c80466d"
server_key_iv: "d587bf90da04048a37b63bfa27612725"
server_key_version: "encryptionkey@0"
I20260806 02:59:02.780174 2959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-1/wal/instance:
uuid: "cb1071102e414a35bd83ae06489d0392"
format_stamp: "Formatted at 2026-08-06 02:59:02 on dist-test-slave-dhph"
server_key: "0dab08749a3c813ce41dc8822c80466d"
server_key_iv: "d587bf90da04048a37b63bfa27612725"
server_key_version: "encryptionkey@0"
I20260806 02:59:02.787045 2959 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260806 02:59:02.791484 2974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:02.793927 2959 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260806 02:59:02.794262 2959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-1/wal
uuid: "cb1071102e414a35bd83ae06489d0392"
format_stamp: "Formatted at 2026-08-06 02:59:02 on dist-test-slave-dhph"
server_key: "0dab08749a3c813ce41dc8822c80466d"
server_key_iv: "d587bf90da04048a37b63bfa27612725"
server_key_version: "encryptionkey@0"
I20260806 02:59:02.794741 2959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:59:02.824256 2959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:59:02.830698 2959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:59:02.831306 2959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:59:02.832664 2959 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:59:02.835253 2959 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:59:02.835346 2959 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:02.835494 2959 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:59:02.835552 2959 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:02.892458 2959 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.194:39351
I20260806 02:59:02.892498 3087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.194:39351 every 8 connection(s)
I20260806 02:59:02.895108 2959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
I20260806 02:59:02.902339 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 2959
I20260806 02:59:02.902806 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-1/wal/instance
I20260806 02:59:02.903661 32043 external_mini_cluster.cc:1428] Setting key 2781225eb016ab16ce37e2a806aa6c47
Aug 06 02:59:02 dist-test-slave-dhph krb5kdc[2717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985142, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.194@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 02:59:02.936978 3090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:02.899652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.194:55673 (local address 127.31.74.254:38741)
0806 02:59:02.900375 (+ 723us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:02.900383 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:02.902574 (+ 2191us) server_negotiation.cc:407] Connection header received
0806 02:59:02.906249 (+ 3675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:02.906256 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:02.906368 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:02.906510 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:02.911385 (+ 4875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:02.912706 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:02.916734 (+ 4028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:02.917613 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:02.918002 (+ 389us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:02.924169 (+ 6167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:02.924176 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:02.924180 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:02.924231 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:02.931547 (+ 7316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:02.934010 (+ 2463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:02.934022 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:02.934025 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:02.934207 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:02.934808 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:02.934822 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:02.934826 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:02.935161 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:02.935372 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:02.936216 (+ 844us) server_negotiation.cc:299] Negotiation successful
0806 02:59:02.936517 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":288,"threads_started":1}
I20260806 02:59:02.938247 3091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:02.900679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:38741 (local address 127.31.74.194:55673)
0806 02:59:02.902282 (+ 1603us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:02.902334 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:02.905856 (+ 3522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:02.907172 (+ 1316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:02.907190 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:02.909155 (+ 1965us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:59:02.910857 (+ 1702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:02.910970 (+ 113us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:02.913232 (+ 2262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:02.913242 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:02.916027 (+ 2785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:02.916066 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:02.918158 (+ 2092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:02.918174 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:02.918808 (+ 634us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:02.920181 (+ 1373us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:59:02.920210 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:59:02.922953 (+ 2743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:59:02.932518 (+ 9565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:02.932528 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:02.932545 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:02.933705 (+ 1160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:02.934351 (+ 646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:02.934359 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:02.934364 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:02.934659 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:02.935321 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:59:02.935330 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:59:02.935768 (+ 438us) client_negotiation.cc:769] Sending connection context
0806 02:59:02.936101 (+ 333us) client_negotiation.cc:240] Negotiation successful
0806 02:59:02.936767 (+ 666us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1335,"thread_start_us":730,"threads_started":1}
I20260806 02:59:02.940372 3088 heartbeater.cc:347] Connected to a master server at 127.31.74.254:38741
I20260806 02:59:02.940754 3088 heartbeater.cc:464] Registering TS with master...
I20260806 02:59:02.942030 3088 heartbeater.cc:511] Master 127.31.74.254:38741 requested a full tablet report, sending...
I20260806 02:59:02.945399 2762 ts_manager.cc:194] Registered new tserver with Master: cb1071102e414a35bd83ae06489d0392 (127.31.74.194:39351)
WARNING: no policy specified for kudu/127.31.74.195@KRBTEST.COM; defaulting to no policy
I20260806 02:59:02.947430 2762 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.74.194@KRBTEST.COM'} at 127.31.74.194:55673
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.195.keytab.
Entry for principal kudu/127.31.74.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.195.keytab.
I20260806 02:59:02.997627 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.195:0
--local_ip_for_outbound_sockets=127.31.74.195
--webserver_interface=127.31.74.195
--webserver_port=0
--tserver_master_addrs=127.31.74.254:38741
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.195.keytab
--principal=kudu/127.31.74.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:36733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:59:03.225800 2953 heartbeater.cc:503] Master 127.31.74.254:38741 was elected leader, sending a full tablet report...
W20260806 02:59:03.527228 3094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:59:03.527653 3094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:59:03.527856 3094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:59:03.537249 3094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:59:03.537495 3094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:59:03.537669 3094 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.195
I20260806 02:59:03.551076 3094 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:36733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.195.keytab
--principal=kudu/127.31.74.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.74.195
--webserver_port=0
--tserver_master_addrs=127.31.74.254:38741
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.195
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:59:03.553716 3094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:59:03.555488 3094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:59:03.574745 3099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:03.574666 3102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:03.574906 3100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:59:03.575664 3094 server_base.cc:1034] running on GCE node
I20260806 02:59:03.576407 3094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:59:03.577656 3094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:59:03.578943 3094 hybrid_clock.cc:648] HybridClock initialized: now 1785985143578891 us; error 37 us; skew 500 ppm
I20260806 02:59:03.579465 3094 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:59:03 dist-test-slave-dhph krb5kdc[2717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985143, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:59:03.583957 3094 init.cc:377] Logged in from keytab as kudu/127.31.74.195@KRBTEST.COM (short username kudu)
I20260806 02:59:03.585913 3094 webserver.cc:460] Webserver started at http://127.31.74.195:34221/ using document root <none> and password file <none>
I20260806 02:59:03.586822 3094 fs_manager.cc:359] Metadata directory not provided
I20260806 02:59:03.586997 3094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:59:03.587375 3094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:59:03.591904 3094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-2/data/instance:
uuid: "ad294e5e2e8c4ab0ad88ff599c580935"
format_stamp: "Formatted at 2026-08-06 02:59:03 on dist-test-slave-dhph"
server_key: "682d49f7b7388573e8291e0ca5abda9a"
server_key_iv: "ec48f811d338647358240dad93249a6a"
server_key_version: "encryptionkey@0"
I20260806 02:59:03.592844 3094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-2/wal/instance:
uuid: "ad294e5e2e8c4ab0ad88ff599c580935"
format_stamp: "Formatted at 2026-08-06 02:59:03 on dist-test-slave-dhph"
server_key: "682d49f7b7388573e8291e0ca5abda9a"
server_key_iv: "ec48f811d338647358240dad93249a6a"
server_key_version: "encryptionkey@0"
I20260806 02:59:03.601594 3094 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.004s
I20260806 02:59:03.608359 3109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:03.612380 3094 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.005s sys 0.001s
I20260806 02:59:03.612670 3094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-2/wal
uuid: "ad294e5e2e8c4ab0ad88ff599c580935"
format_stamp: "Formatted at 2026-08-06 02:59:03 on dist-test-slave-dhph"
server_key: "682d49f7b7388573e8291e0ca5abda9a"
server_key_iv: "ec48f811d338647358240dad93249a6a"
server_key_version: "encryptionkey@0"
I20260806 02:59:03.612968 3094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:59:03.633167 3094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:59:03.641083 3094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:59:03.641472 3094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:59:03.643038 3094 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:59:03.644906 3094 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:59:03.645037 3094 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:03.645210 3094 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:59:03.645291 3094 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:03.704931 3094 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.195:39703
I20260806 02:59:03.704972 3221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.195:39703 every 8 connection(s)
I20260806 02:59:03.706414 3094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
I20260806 02:59:03.713428 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 3094
I20260806 02:59:03.713872 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785985105255475-32043-0/minicluster-data/ts-2/wal/instance
I20260806 02:59:03.714718 32043 external_mini_cluster.cc:1428] Setting key 420763dd9d12af59c20334268f81f0b0
Aug 06 02:59:03 dist-test-slave-dhph krb5kdc[2717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985143, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.195@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 02:59:03.740260 3224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:03.710526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.195:38667 (local address 127.31.74.254:38741)
0806 02:59:03.711447 (+ 921us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:03.711455 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:03.713724 (+ 2269us) server_negotiation.cc:407] Connection header received
0806 02:59:03.716634 (+ 2910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:03.716639 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:03.716732 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:03.716892 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:03.719081 (+ 2189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:03.720027 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:03.722833 (+ 2806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:03.723903 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:03.724294 (+ 391us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:03.731784 (+ 7490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:03.731793 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:03.731800 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:03.731857 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:03.736141 (+ 4284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:03.737204 (+ 1063us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:03.737210 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:03.737212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:03.737372 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:03.738048 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:03.738052 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:03.738055 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:03.738434 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:03.738618 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:03.739711 (+ 1093us) server_negotiation.cc:299] Negotiation successful
0806 02:59:03.739941 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":736,"thread_start_us":569,"threads_started":1}
I20260806 02:59:03.741245 3225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:03.711663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:38741 (local address 127.31.74.195:38667)
0806 02:59:03.713365 (+ 1702us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:03.713431 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:03.716229 (+ 2798us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:03.717173 (+ 944us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:03.717183 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:03.717939 (+ 756us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:59:03.718744 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:03.718781 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:03.720542 (+ 1761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:03.720548 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:03.722273 (+ 1725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:03.722291 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:03.724466 (+ 2175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:03.724473 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:03.725298 (+ 825us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:03.726486 (+ 1188us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:59:03.726518 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:59:03.731373 (+ 4855us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:59:03.736364 (+ 4991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:03.736374 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:03.736392 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:03.737019 (+ 627us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:03.737623 (+ 604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:03.737627 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:03.737629 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:03.737872 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:03.738773 (+ 901us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:59:03.738781 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:59:03.739241 (+ 460us) client_negotiation.cc:769] Sending connection context
0806 02:59:03.739525 (+ 284us) client_negotiation.cc:240] Negotiation successful
0806 02:59:03.740011 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1050,"thread_start_us":660,"threads_started":1}
I20260806 02:59:03.743335 3222 heartbeater.cc:347] Connected to a master server at 127.31.74.254:38741
I20260806 02:59:03.744032 3222 heartbeater.cc:464] Registering TS with master...
I20260806 02:59:03.746119 3222 heartbeater.cc:511] Master 127.31.74.254:38741 requested a full tablet report, sending...
I20260806 02:59:03.749493 2762 ts_manager.cc:194] Registered new tserver with Master: ad294e5e2e8c4ab0ad88ff599c580935 (127.31.74.195:39703)
I20260806 02:59:03.750994 2762 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.74.195@KRBTEST.COM'} at 127.31.74.195:38667
I20260806 02:59:03.761834 32043 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 02:59:03 dist-test-slave-dhph krb5kdc[2717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985143, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 06 02:59:03 dist-test-slave-dhph krb5kdc[2717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985143, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 02:59:03.838743 3224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:03.818553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60530 (local address 127.31.74.254:38741)
0806 02:59:03.818809 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:03.818815 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:03.818865 (+ 50us) server_negotiation.cc:407] Connection header received
0806 02:59:03.819109 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:03.819114 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:03.819258 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:03.819387 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:03.821234 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:03.822076 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:03.823754 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:03.824324 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:03.824650 (+ 326us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:03.830169 (+ 5519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:03.830176 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:03.830180 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:03.830225 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:03.834199 (+ 3974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:03.835624 (+ 1425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:03.835632 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:03.835634 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:03.835883 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:03.836481 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:03.836485 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:03.836487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:03.836929 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:03.837226 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:03.837659 (+ 433us) server_negotiation.cc:299] Negotiation successful
0806 02:59:03.838166 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
W20260806 02:59:03.839532 2762 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:60530
I20260806 02:59:03.842021 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 2824
Aug 06 02:59:03 dist-test-slave-dhph krb5kdc[2717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985143, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.31.74.193@KRBTEST.COM
I20260806 02:59:03.868994 3234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:03.843355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33700 (local address 127.31.74.193:36901)
0806 02:59:03.844096 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:03.844104 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:03.844140 (+ 36us) server_negotiation.cc:407] Connection header received
0806 02:59:03.844205 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:03.844211 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:03.844443 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:03.844689 (+ 246us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:03.846090 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:03.848180 (+ 2090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:03.850927 (+ 2747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:03.851652 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:03.851797 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:03.857433 (+ 5636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:03.857454 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:03.857472 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:03.857522 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:03.863479 (+ 5957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:03.864740 (+ 1261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:03.864756 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:03.864759 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:03.864944 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:03.865802 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:03.865812 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:03.865815 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:03.866282 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:03.866729 (+ 447us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:03.867354 (+ 625us) server_negotiation.cc:299] Negotiation successful
0806 02:59:03.867703 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":394,"threads_started":1}
W20260806 02:59:03.870801 2948 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:33700
W20260806 02:59:03.872572 32043 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 2824: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 02:59:03.872906 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 2824
I20260806 02:59:03.901222 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 2959
Aug 06 02:59:03 dist-test-slave-dhph krb5kdc[2717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985143, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.31.74.194@KRBTEST.COM
I20260806 02:59:03.934065 3235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:03.903150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42484 (local address 127.31.74.194:39351)
0806 02:59:03.904049 (+ 899us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:03.904056 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:03.904085 (+ 29us) server_negotiation.cc:407] Connection header received
0806 02:59:03.904149 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:03.904164 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:03.904376 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:03.904532 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:03.907526 (+ 2994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:03.909091 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:03.912635 (+ 3544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:03.913398 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:03.913550 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:03.920985 (+ 7435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:03.921005 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:03.921024 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:03.921081 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:03.926571 (+ 5490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:03.929038 (+ 2467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:03.929049 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:03.929052 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:03.930892 (+ 1840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:03.931818 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:03.931823 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:03.931826 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:03.932313 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:03.932604 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:03.933317 (+ 713us) server_negotiation.cc:299] Negotiation successful
0806 02:59:03.933715 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":732,"thread_start_us":347,"threads_started":1}
W20260806 02:59:03.935779 3083 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:42484
W20260806 02:59:03.937714 32043 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 2959: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 02:59:03.937893 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 2959
I20260806 02:59:03.967160 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 3094
Aug 06 02:59:03 dist-test-slave-dhph krb5kdc[2717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985143, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.31.74.195@KRBTEST.COM
I20260806 02:59:03.994796 3236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:03.969154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45914 (local address 127.31.74.195:39703)
0806 02:59:03.970550 (+ 1396us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:03.970558 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:03.970992 (+ 434us) server_negotiation.cc:407] Connection header received
0806 02:59:03.971387 (+ 395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:03.971400 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:03.971677 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:03.971899 (+ 222us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:03.973539 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:03.975050 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:03.976799 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:03.977495 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:03.977669 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:03.985429 (+ 7760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:03.985456 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:03.985475 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:03.985529 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:03.989793 (+ 4264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:03.991079 (+ 1286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:03.991096 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:03.991099 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:03.991281 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:03.991874 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:03.991880 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:03.991885 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:03.992651 (+ 766us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:03.993531 (+ 880us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:03.993926 (+ 395us) server_negotiation.cc:299] Negotiation successful
0806 02:59:03.994273 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1219,"thread_start_us":421,"threads_started":1}
W20260806 02:59:03.997246 3217 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:45914
W20260806 02:59:04.000039 32043 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 3094: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 02:59:04.000250 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 3094
I20260806 02:59:04.025919 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 2730
I20260806 02:59:04.029055 2793 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:59:04.168365 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 2730
2026-08-06T02:59:04Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (4995 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:59:04 dist-test-slave-dhph krb5kdc[3241](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:59:04 dist-test-slave-dhph krb5kdc[3241](info): set up 2 sockets
Aug 06 02:59:04 dist-test-slave-dhph krb5kdc[3241](info): commencing operation
krb5kdc: starting...
W20260806 02:59:05.600991 32043 mini_kdc.cc:121] Time spent starting KDC: real 1.373s user 0.004s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/krb5kdc/test-user.keytab.
Aug 06 02:59:05 dist-test-slave-dhph krb5kdc[3241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:59:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:59:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.74.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab.
Entry for principal kudu/127.31.74.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab.
I20260806 02:59:05.936256 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:38781
--webserver_interface=127.31.74.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab
--principal=kudu/127.31.74.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:39501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.74.254:38781
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:59:06.395876 3254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:59:06.396334 3254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:59:06.396807 3254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:59:06.407076 3254 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:59:06.407223 3254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:59:06.407274 3254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:59:06.407373 3254 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:59:06.407444 3254 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:59:06.421419 3254 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:39501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.74.254:38781
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab
--principal=kudu/127.31.74.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:38781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.74.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:59:06.423075 3254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:59:06.424975 3254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:59:06.441792 3260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:06.442281 3262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:06.442317 3259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:59:06.449589 3254 server_base.cc:1034] running on GCE node
I20260806 02:59:06.450734 3254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:59:06.452527 3254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:59:06.454133 3254 hybrid_clock.cc:648] HybridClock initialized: now 1785985146453998 us; error 83 us; skew 500 ppm
I20260806 02:59:06.454864 3254 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:59:06 dist-test-slave-dhph krb5kdc[3241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985146, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:59:06.463021 3254 init.cc:377] Logged in from keytab as kudu/127.31.74.254@KRBTEST.COM (short username kudu)
I20260806 02:59:06.465255 3254 webserver.cc:460] Webserver started at http://127.31.74.254:33413/ using document root <none> and password file <none>
I20260806 02:59:06.466125 3254 fs_manager.cc:359] Metadata directory not provided
I20260806 02:59:06.466311 3254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:59:06.466790 3254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:59:06.470471 3254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/minicluster-data/master-0/data/instance:
uuid: "01536e585772486eb490cecfd687dd05"
format_stamp: "Formatted at 2026-08-06 02:59:06 on dist-test-slave-dhph"
server_key: "449f9cc23da1b9cce8d00b0242e0378e"
server_key_iv: "eeeba1dc88aae3a1aab786bde241fb0a"
server_key_version: "encryptionkey@0"
I20260806 02:59:06.471487 3254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/minicluster-data/master-0/wal/instance:
uuid: "01536e585772486eb490cecfd687dd05"
format_stamp: "Formatted at 2026-08-06 02:59:06 on dist-test-slave-dhph"
server_key: "449f9cc23da1b9cce8d00b0242e0378e"
server_key_iv: "eeeba1dc88aae3a1aab786bde241fb0a"
server_key_version: "encryptionkey@0"
I20260806 02:59:06.477603 3254 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260806 02:59:06.482874 3269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:06.485244 3254 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260806 02:59:06.485517 3254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/minicluster-data/master-0/wal
uuid: "01536e585772486eb490cecfd687dd05"
format_stamp: "Formatted at 2026-08-06 02:59:06 on dist-test-slave-dhph"
server_key: "449f9cc23da1b9cce8d00b0242e0378e"
server_key_iv: "eeeba1dc88aae3a1aab786bde241fb0a"
server_key_version: "encryptionkey@0"
I20260806 02:59:06.485806 3254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:59:06.514219 3254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:59:06.521238 3254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:59:06.521721 3254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:59:06.550109 3254 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.254:38781
I20260806 02:59:06.550372 3323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.254:38781 every 8 connection(s)
I20260806 02:59:06.551818 3254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
I20260806 02:59:06.557384 3324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:59:06.561652 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 3254
I20260806 02:59:06.561947 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/minicluster-data/master-0/wal/instance
I20260806 02:59:06.562875 32043 external_mini_cluster.cc:1428] Setting key 6eb5b6e8178b93e6c2fa212868ca1da4
I20260806 02:59:06.575765 3324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01536e585772486eb490cecfd687dd05: Bootstrap starting.
I20260806 02:59:06.581302 3324 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 01536e585772486eb490cecfd687dd05: Neither blocks nor log segments found. Creating new log.
I20260806 02:59:06.583180 3324 log.cc:824] T 00000000000000000000000000000000 P 01536e585772486eb490cecfd687dd05: Log is configured to *not* fsync() on all Append() calls
Aug 06 02:59:06 dist-test-slave-dhph krb5kdc[3241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 02:59:06.587572 3324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01536e585772486eb490cecfd687dd05: No bootstrap required, opened a new log
I20260806 02:59:06.601488 3324 raft_consensus.cc:348] T 00000000000000000000000000000000 P 01536e585772486eb490cecfd687dd05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01536e585772486eb490cecfd687dd05" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 38781 } }
I20260806 02:59:06.602104 3324 raft_consensus.cc:374] T 00000000000000000000000000000000 P 01536e585772486eb490cecfd687dd05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:59:06.602555 3324 raft_consensus.cc:729] T 00000000000000000000000000000000 P 01536e585772486eb490cecfd687dd05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01536e585772486eb490cecfd687dd05, State: Initialized, Role: FOLLOWER
I20260806 02:59:06.602737 3327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:06.565625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54210 (local address 127.31.74.254:38781)
0806 02:59:06.567221 (+ 1596us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:06.567236 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:06.567286 (+ 50us) server_negotiation.cc:407] Connection header received
0806 02:59:06.569525 (+ 2239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:06.569557 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:06.570086 (+ 529us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:06.570514 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:06.573149 (+ 2635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:06.575732 (+ 2583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:06.578360 (+ 2628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:06.579762 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:06.579944 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:06.586825 (+ 6881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:06.586838 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:06.586861 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:06.586919 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:06.592790 (+ 5871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:06.594331 (+ 1541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:06.594344 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:06.594363 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:06.594716 (+ 353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:06.595228 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:06.595239 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:06.595244 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:06.596199 (+ 955us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:06.596541 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:06.597966 (+ 1425us) server_negotiation.cc:299] Negotiation successful
0806 02:59:06.599165 (+ 1199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1356,"thread_start_us":472,"threads_started":1}
I20260806 02:59:06.604094 3324 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01536e585772486eb490cecfd687dd05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01536e585772486eb490cecfd687dd05" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 38781 } }
I20260806 02:59:06.605114 3324 raft_consensus.cc:388] T 00000000000000000000000000000000 P 01536e585772486eb490cecfd687dd05 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:59:06.605643 3324 raft_consensus.cc:482] T 00000000000000000000000000000000 P 01536e585772486eb490cecfd687dd05 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:59:06.606201 3324 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 01536e585772486eb490cecfd687dd05 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:59:06.611668 3324 raft_consensus.cc:504] T 00000000000000000000000000000000 P 01536e585772486eb490cecfd687dd05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01536e585772486eb490cecfd687dd05" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 38781 } }
I20260806 02:59:06.613022 3324 leader_election.cc:302] T 00000000000000000000000000000000 P 01536e585772486eb490cecfd687dd05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 01536e585772486eb490cecfd687dd05; no voters:
I20260806 02:59:06.614145 3324 leader_election.cc:288] T 00000000000000000000000000000000 P 01536e585772486eb490cecfd687dd05 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:59:06.614328 3329 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 01536e585772486eb490cecfd687dd05 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:59:06.615173 3329 raft_consensus.cc:686] T 00000000000000000000000000000000 P 01536e585772486eb490cecfd687dd05 [term 1 LEADER]: Becoming Leader. State: Replica: 01536e585772486eb490cecfd687dd05, State: Running, Role: LEADER
I20260806 02:59:06.616179 3329 consensus_queue.cc:237] T 00000000000000000000000000000000 P 01536e585772486eb490cecfd687dd05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01536e585772486eb490cecfd687dd05" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 38781 } }
I20260806 02:59:06.617529 3324 sys_catalog.cc:564] T 00000000000000000000000000000000 P 01536e585772486eb490cecfd687dd05 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:59:06.620746 3331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01536e585772486eb490cecfd687dd05 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01536e585772486eb490cecfd687dd05. Latest consensus state: current_term: 1 leader_uuid: "01536e585772486eb490cecfd687dd05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01536e585772486eb490cecfd687dd05" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 38781 } } }
I20260806 02:59:06.621238 3331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01536e585772486eb490cecfd687dd05 [sys.catalog]: This master's current role is: LEADER
I20260806 02:59:06.623065 3330 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01536e585772486eb490cecfd687dd05 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "01536e585772486eb490cecfd687dd05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01536e585772486eb490cecfd687dd05" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 38781 } } }
I20260806 02:59:06.625741 3330 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01536e585772486eb490cecfd687dd05 [sys.catalog]: This master's current role is: LEADER
I20260806 02:59:06.628764 3337 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:59:06.639137 3337 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:59:06.657023 3337 catalog_manager.cc:1347] Generated new cluster ID: 801985c0b5434e1981ee8a603d9c77b8
I20260806 02:59:06.657552 3337 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:59:06.694487 3337 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:59:06.696687 3337 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:59:06.714331 3337 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 01536e585772486eb490cecfd687dd05: Generated new TSK 0
I20260806 02:59:06.716576 3337 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.74.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.193.keytab.
Entry for principal kudu/127.31.74.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.193.keytab.
I20260806 02:59:06.826995 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.193:0
--local_ip_for_outbound_sockets=127.31.74.193
--webserver_interface=127.31.74.193
--webserver_port=0
--tserver_master_addrs=127.31.74.254:38781
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.193.keytab
--principal=kudu/127.31.74.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:39501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:59:07.305419 3351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:59:07.305785 3351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:59:07.305992 3351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:59:07.316483 3351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:59:07.316668 3351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:59:07.316872 3351 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.193
I20260806 02:59:07.329164 3351 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:39501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.193.keytab
--principal=kudu/127.31.74.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.74.193
--webserver_port=0
--tserver_master_addrs=127.31.74.254:38781
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.193
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:59:07.330734 3351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:59:07.333586 3351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:59:07.353489 3359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:07.353489 3357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:07.353988 3356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:59:07.356887 3351 server_base.cc:1034] running on GCE node
I20260806 02:59:07.357697 3351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:59:07.359124 3351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:59:07.360481 3351 hybrid_clock.cc:648] HybridClock initialized: now 1785985147360397 us; error 71 us; skew 500 ppm
I20260806 02:59:07.361090 3351 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:59:07 dist-test-slave-dhph krb5kdc[3241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985147, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:59:07.366153 3351 init.cc:377] Logged in from keytab as kudu/127.31.74.193@KRBTEST.COM (short username kudu)
I20260806 02:59:07.368561 3351 webserver.cc:460] Webserver started at http://127.31.74.193:33969/ using document root <none> and password file <none>
I20260806 02:59:07.369668 3351 fs_manager.cc:359] Metadata directory not provided
I20260806 02:59:07.370144 3351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:59:07.370947 3351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:59:07.375337 3351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/minicluster-data/ts-0/data/instance:
uuid: "fbb9f033ecea49fb8fb6d2a3eab3f1a5"
format_stamp: "Formatted at 2026-08-06 02:59:07 on dist-test-slave-dhph"
server_key: "0b97abcd1deca0ff0fa61dc21494a39f"
server_key_iv: "2520922993f7867ff7883e2560b34fe1"
server_key_version: "encryptionkey@0"
I20260806 02:59:07.376729 3351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/minicluster-data/ts-0/wal/instance:
uuid: "fbb9f033ecea49fb8fb6d2a3eab3f1a5"
format_stamp: "Formatted at 2026-08-06 02:59:07 on dist-test-slave-dhph"
server_key: "0b97abcd1deca0ff0fa61dc21494a39f"
server_key_iv: "2520922993f7867ff7883e2560b34fe1"
server_key_version: "encryptionkey@0"
I20260806 02:59:07.384469 3351 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.005s
I20260806 02:59:07.390748 3367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:07.393898 3351 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.001s
I20260806 02:59:07.394181 3351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/minicluster-data/ts-0/wal
uuid: "fbb9f033ecea49fb8fb6d2a3eab3f1a5"
format_stamp: "Formatted at 2026-08-06 02:59:07 on dist-test-slave-dhph"
server_key: "0b97abcd1deca0ff0fa61dc21494a39f"
server_key_iv: "2520922993f7867ff7883e2560b34fe1"
server_key_version: "encryptionkey@0"
I20260806 02:59:07.394527 3351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:59:07.422490 3351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:59:07.430116 3351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:59:07.430883 3351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:59:07.433522 3351 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:59:07.435511 3351 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:59:07.435664 3351 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:07.435853 3351 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:59:07.435935 3351 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:07.488526 3351 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.193:42163
I20260806 02:59:07.488590 3482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.193:42163 every 8 connection(s)
I20260806 02:59:07.490168 3351 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
I20260806 02:59:07.491931 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 3351
I20260806 02:59:07.492235 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/minicluster-data/ts-0/wal/instance
I20260806 02:59:07.493625 32043 external_mini_cluster.cc:1428] Setting key 21bd81e737c68ad5258c37e83ebe89b5
Aug 06 02:59:07 dist-test-slave-dhph krb5kdc[3241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985147, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.193@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 02:59:07.522392 3485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:07.495124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.193:53643 (local address 127.31.74.254:38781)
0806 02:59:07.495684 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:07.495690 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:07.496909 (+ 1219us) server_negotiation.cc:407] Connection header received
0806 02:59:07.499107 (+ 2198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:07.499113 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:07.499216 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:07.499366 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:07.501906 (+ 2540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:07.503341 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:07.507768 (+ 4427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:07.508672 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:07.508869 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:07.514961 (+ 6092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:07.514969 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:07.514975 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:07.515030 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:07.518790 (+ 3760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:07.519836 (+ 1046us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:07.519841 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:07.519844 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:07.519973 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:07.520457 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:07.520466 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:07.520468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:07.520761 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:07.520936 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:07.521705 (+ 769us) server_negotiation.cc:299] Negotiation successful
0806 02:59:07.521958 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":306,"threads_started":1}
WARNING: no policy specified for kudu/127.31.74.194@KRBTEST.COM; defaulting to no policy
I20260806 02:59:07.523413 3486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:07.495705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:38781 (local address 127.31.74.193:53643)
0806 02:59:07.496720 (+ 1015us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:07.496766 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:07.498810 (+ 2044us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:07.499668 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:07.499683 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:07.500636 (+ 953us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:59:07.501630 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:07.501659 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:07.503542 (+ 1883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:07.503728 (+ 186us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:07.507413 (+ 3685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:07.507441 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:07.509201 (+ 1760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:07.509212 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:07.509689 (+ 477us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:07.510843 (+ 1154us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:59:07.510873 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:59:07.514549 (+ 3676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:59:07.518986 (+ 4437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:07.518997 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:07.519018 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:07.519671 (+ 653us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:07.520098 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:07.520104 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:07.520107 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:07.520323 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:07.520908 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:59:07.520917 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:59:07.521312 (+ 395us) client_negotiation.cc:769] Sending connection context
0806 02:59:07.521594 (+ 282us) client_negotiation.cc:240] Negotiation successful
0806 02:59:07.522222 (+ 628us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":742,"thread_start_us":465,"threads_started":1}
I20260806 02:59:07.525920 3483 heartbeater.cc:347] Connected to a master server at 127.31.74.254:38781
I20260806 02:59:07.526321 3483 heartbeater.cc:464] Registering TS with master...
I20260806 02:59:07.527549 3483 heartbeater.cc:511] Master 127.31.74.254:38781 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.194@KRBTEST.COM" created.
I20260806 02:59:07.531116 3287 ts_manager.cc:194] Registered new tserver with Master: fbb9f033ecea49fb8fb6d2a3eab3f1a5 (127.31.74.193:42163)
I20260806 02:59:07.534209 3287 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.74.193@KRBTEST.COM'} at 127.31.74.193:53643
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.194.keytab.
Entry for principal kudu/127.31.74.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.194.keytab.
I20260806 02:59:07.568344 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.194:0
--local_ip_for_outbound_sockets=127.31.74.194
--webserver_interface=127.31.74.194
--webserver_port=0
--tserver_master_addrs=127.31.74.254:38781
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.194.keytab
--principal=kudu/127.31.74.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:39501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:59:07.992486 3489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:59:07.992821 3489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:59:07.992977 3489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:59:08.001768 3489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:59:08.001907 3489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:59:08.002030 3489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.194
I20260806 02:59:08.014139 3489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:39501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.194.keytab
--principal=kudu/127.31.74.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.74.194
--webserver_port=0
--tserver_master_addrs=127.31.74.254:38781
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.194
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:59:08.015760 3489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:59:08.017373 3489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:59:08.034543 3495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:08.034608 3497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:08.034658 3494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:59:08.035694 3489 server_base.cc:1034] running on GCE node
I20260806 02:59:08.036994 3489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:59:08.038751 3489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:59:08.040030 3489 hybrid_clock.cc:648] HybridClock initialized: now 1785985148039944 us; error 80 us; skew 500 ppm
I20260806 02:59:08.040434 3489 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:59:08 dist-test-slave-dhph krb5kdc[3241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985148, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:59:08.044397 3489 init.cc:377] Logged in from keytab as kudu/127.31.74.194@KRBTEST.COM (short username kudu)
I20260806 02:59:08.046119 3489 webserver.cc:460] Webserver started at http://127.31.74.194:44523/ using document root <none> and password file <none>
I20260806 02:59:08.047017 3489 fs_manager.cc:359] Metadata directory not provided
I20260806 02:59:08.047152 3489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:59:08.047461 3489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:59:08.051086 3489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/minicluster-data/ts-1/data/instance:
uuid: "74457bda26454849a3f9d94ca0d8c5c9"
format_stamp: "Formatted at 2026-08-06 02:59:08 on dist-test-slave-dhph"
server_key: "cad3d04401c0ec9fe647f6a9370fa0e3"
server_key_iv: "bd790bf2f22b3eb970164121bcb9962f"
server_key_version: "encryptionkey@0"
I20260806 02:59:08.051975 3489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/minicluster-data/ts-1/wal/instance:
uuid: "74457bda26454849a3f9d94ca0d8c5c9"
format_stamp: "Formatted at 2026-08-06 02:59:08 on dist-test-slave-dhph"
server_key: "cad3d04401c0ec9fe647f6a9370fa0e3"
server_key_iv: "bd790bf2f22b3eb970164121bcb9962f"
server_key_version: "encryptionkey@0"
I20260806 02:59:08.058791 3489 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260806 02:59:08.065935 3504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:08.068321 3489 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260806 02:59:08.068572 3489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/minicluster-data/ts-1/wal
uuid: "74457bda26454849a3f9d94ca0d8c5c9"
format_stamp: "Formatted at 2026-08-06 02:59:08 on dist-test-slave-dhph"
server_key: "cad3d04401c0ec9fe647f6a9370fa0e3"
server_key_iv: "bd790bf2f22b3eb970164121bcb9962f"
server_key_version: "encryptionkey@0"
I20260806 02:59:08.068854 3489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:59:08.086016 3489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:59:08.091424 3489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:59:08.091747 3489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:59:08.092844 3489 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:59:08.094359 3489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:59:08.094648 3489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:08.094841 3489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:59:08.094924 3489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:08.147033 3489 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.194:36201
I20260806 02:59:08.147131 3618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.194:36201 every 8 connection(s)
I20260806 02:59:08.149084 3489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
I20260806 02:59:08.154013 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 3489
I20260806 02:59:08.154259 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/minicluster-data/ts-1/wal/instance
I20260806 02:59:08.154944 32043 external_mini_cluster.cc:1428] Setting key e0f9fa6e2beac6b5cc6ddc831d258ac9
Aug 06 02:59:08 dist-test-slave-dhph krb5kdc[3241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985148, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.194@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 02:59:08.189847 3621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:08.152558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.194:39363 (local address 127.31.74.254:38781)
0806 02:59:08.153195 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:08.153200 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:08.154540 (+ 1340us) server_negotiation.cc:407] Connection header received
0806 02:59:08.159325 (+ 4785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:08.159442 (+ 117us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:08.159572 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:08.159699 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:08.162447 (+ 2748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:08.163927 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:08.167757 (+ 3830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:08.168263 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:08.168431 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:08.176767 (+ 8336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:08.176778 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:08.176796 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:08.176860 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:08.184312 (+ 7452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:08.185656 (+ 1344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:08.185666 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:08.185673 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:08.186009 (+ 336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:08.186844 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:08.186849 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:08.186851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:08.187218 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:08.187367 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:08.188802 (+ 1435us) server_negotiation.cc:299] Negotiation successful
0806 02:59:08.189401 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":418,"threads_started":1}
I20260806 02:59:08.191256 3623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:08.153156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:38781 (local address 127.31.74.194:39363)
0806 02:59:08.154301 (+ 1145us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:08.154394 (+ 93us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:08.156821 (+ 2427us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:08.160071 (+ 3250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:08.160086 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:08.161033 (+ 947us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:59:08.162117 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:08.162152 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:08.164192 (+ 2040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:08.164201 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:08.167222 (+ 3021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:08.167256 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:08.168928 (+ 1672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:08.169004 (+ 76us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:08.169565 (+ 561us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:08.170866 (+ 1301us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:59:08.170912 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:59:08.176127 (+ 5215us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:59:08.184606 (+ 8479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:08.184618 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:08.184636 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:08.185282 (+ 646us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:08.186194 (+ 912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:08.186203 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:08.186210 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:08.186617 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:08.187402 (+ 785us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:59:08.187416 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:59:08.188270 (+ 854us) client_negotiation.cc:769] Sending connection context
0806 02:59:08.188646 (+ 376us) client_negotiation.cc:240] Negotiation successful
0806 02:59:08.189366 (+ 720us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":816,"thread_start_us":567,"threads_started":1}
I20260806 02:59:08.193512 3619 heartbeater.cc:347] Connected to a master server at 127.31.74.254:38781
I20260806 02:59:08.194031 3619 heartbeater.cc:464] Registering TS with master...
I20260806 02:59:08.194948 3619 heartbeater.cc:511] Master 127.31.74.254:38781 requested a full tablet report, sending...
I20260806 02:59:08.197598 3287 ts_manager.cc:194] Registered new tserver with Master: 74457bda26454849a3f9d94ca0d8c5c9 (127.31.74.194:36201)
WARNING: no policy specified for kudu/127.31.74.195@KRBTEST.COM; defaulting to no policy
I20260806 02:59:08.200775 3287 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.74.194@KRBTEST.COM'} at 127.31.74.194:39363
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.195.keytab.
Entry for principal kudu/127.31.74.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.195.keytab.
I20260806 02:59:08.248682 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.195:0
--local_ip_for_outbound_sockets=127.31.74.195
--webserver_interface=127.31.74.195
--webserver_port=0
--tserver_master_addrs=127.31.74.254:38781
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.195.keytab
--principal=kudu/127.31.74.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:39501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:59:08.539997 3483 heartbeater.cc:503] Master 127.31.74.254:38781 was elected leader, sending a full tablet report...
W20260806 02:59:08.694409 3626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:59:08.694854 3626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:59:08.695031 3626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:59:08.703246 3626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:59:08.703423 3626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:59:08.703627 3626 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.195
I20260806 02:59:08.713949 3626 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:39501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.195.keytab
--principal=kudu/127.31.74.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.74.195
--webserver_port=0
--tserver_master_addrs=127.31.74.254:38781
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.195
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:59:08.715482 3626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:59:08.716881 3626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:59:08.732033 3632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:08.732391 3635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:08.732409 3633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:59:08.734407 3626 server_base.cc:1034] running on GCE node
I20260806 02:59:08.735733 3626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:59:08.737032 3626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:59:08.738312 3626 hybrid_clock.cc:648] HybridClock initialized: now 1785985148738260 us; error 40 us; skew 500 ppm
I20260806 02:59:08.738801 3626 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:59:08 dist-test-slave-dhph krb5kdc[3241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985148, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:59:08.742995 3626 init.cc:377] Logged in from keytab as kudu/127.31.74.195@KRBTEST.COM (short username kudu)
I20260806 02:59:08.744903 3626 webserver.cc:460] Webserver started at http://127.31.74.195:38103/ using document root <none> and password file <none>
I20260806 02:59:08.745668 3626 fs_manager.cc:359] Metadata directory not provided
I20260806 02:59:08.745837 3626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:59:08.746179 3626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:59:08.749713 3626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/minicluster-data/ts-2/data/instance:
uuid: "a4133ac95ad2489fb43b752f94496de4"
format_stamp: "Formatted at 2026-08-06 02:59:08 on dist-test-slave-dhph"
server_key: "97d50ed63a10a38f42947b9c9d602c9b"
server_key_iv: "371d05722fff2009ba6a518213b72741"
server_key_version: "encryptionkey@0"
I20260806 02:59:08.750762 3626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/minicluster-data/ts-2/wal/instance:
uuid: "a4133ac95ad2489fb43b752f94496de4"
format_stamp: "Formatted at 2026-08-06 02:59:08 on dist-test-slave-dhph"
server_key: "97d50ed63a10a38f42947b9c9d602c9b"
server_key_iv: "371d05722fff2009ba6a518213b72741"
server_key_version: "encryptionkey@0"
I20260806 02:59:08.757812 3626 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260806 02:59:08.762977 3642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:08.764943 3626 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 02:59:08.765202 3626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/minicluster-data/ts-2/wal
uuid: "a4133ac95ad2489fb43b752f94496de4"
format_stamp: "Formatted at 2026-08-06 02:59:08 on dist-test-slave-dhph"
server_key: "97d50ed63a10a38f42947b9c9d602c9b"
server_key_iv: "371d05722fff2009ba6a518213b72741"
server_key_version: "encryptionkey@0"
I20260806 02:59:08.765465 3626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:59:08.783636 3626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:59:08.790256 3626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:59:08.790695 3626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:59:08.792249 3626 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:59:08.794028 3626 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:59:08.794157 3626 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:08.794452 3626 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:59:08.794571 3626 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:08.845005 3626 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.195:42821
I20260806 02:59:08.845052 3756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.195:42821 every 8 connection(s)
I20260806 02:59:08.846484 3626 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
I20260806 02:59:08.854724 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 3626
I20260806 02:59:08.854987 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785985105255475-32043-0/minicluster-data/ts-2/wal/instance
I20260806 02:59:08.855895 32043 external_mini_cluster.cc:1428] Setting key bdff24fc103a89a568be51b6b74a06b1
Aug 06 02:59:08 dist-test-slave-dhph krb5kdc[3241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985148, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.195@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 02:59:08.875247 3759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:08.850851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.195:46231 (local address 127.31.74.254:38781)
0806 02:59:08.851636 (+ 785us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:08.851642 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:08.853472 (+ 1830us) server_negotiation.cc:407] Connection header received
0806 02:59:08.856370 (+ 2898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:08.856379 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:08.856492 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:08.856639 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:08.858942 (+ 2303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:08.859879 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:08.861762 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:08.862253 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:08.862429 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:08.866771 (+ 4342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:08.866777 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:08.866781 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:08.866818 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:08.870356 (+ 3538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:08.872232 (+ 1876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:08.872239 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:08.872241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:08.872405 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:08.873050 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:08.873057 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:08.873061 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:08.873371 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:08.873553 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:08.874422 (+ 869us) server_negotiation.cc:299] Negotiation successful
0806 02:59:08.874738 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":418,"threads_started":1}
I20260806 02:59:08.876446 3760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:08.851827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:38781 (local address 127.31.74.195:46231)
0806 02:59:08.853180 (+ 1353us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:08.853245 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:08.855906 (+ 2661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:08.856951 (+ 1045us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:08.856961 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:08.857737 (+ 776us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:59:08.858549 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:08.858571 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:08.860074 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:08.860080 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:08.861584 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:08.861599 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:08.862540 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:08.862552 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:08.862883 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:08.863740 (+ 857us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:59:08.863766 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:59:08.866407 (+ 2641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:59:08.870568 (+ 4161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:08.870626 (+ 58us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:08.870646 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:08.871884 (+ 1238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:08.872603 (+ 719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:08.872610 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:08.872614 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:08.872899 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:08.873519 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:59:08.873528 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:59:08.873953 (+ 425us) client_negotiation.cc:769] Sending connection context
0806 02:59:08.874332 (+ 379us) client_negotiation.cc:240] Negotiation successful
0806 02:59:08.874962 (+ 630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":882,"thread_start_us":599,"threads_started":1}
I20260806 02:59:08.878512 3757 heartbeater.cc:347] Connected to a master server at 127.31.74.254:38781
I20260806 02:59:08.878907 3757 heartbeater.cc:464] Registering TS with master...
I20260806 02:59:08.879699 3757 heartbeater.cc:511] Master 127.31.74.254:38781 requested a full tablet report, sending...
I20260806 02:59:08.881398 3287 ts_manager.cc:194] Registered new tserver with Master: a4133ac95ad2489fb43b752f94496de4 (127.31.74.195:42821)
I20260806 02:59:08.882423 3287 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.74.195@KRBTEST.COM'} at 127.31.74.195:46231
I20260806 02:59:08.890975 32043 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 02:59:08 dist-test-slave-dhph krb5kdc[3241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 02:59:08 dist-test-slave-dhph krb5kdc[3241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.74.193@KRBTEST.COM
I20260806 02:59:08.969807 3765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:08.948259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44198 (local address 127.31.74.193:42163)
0806 02:59:08.949074 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:08.949083 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:08.949128 (+ 45us) server_negotiation.cc:407] Connection header received
0806 02:59:08.949314 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:08.949322 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:08.949576 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:08.949763 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:08.951490 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:08.952805 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:08.954553 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:08.955457 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:08.955683 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:08.962529 (+ 6846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:08.962550 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:08.962567 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:08.962689 (+ 122us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:08.966472 (+ 3783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:08.967437 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:08.967445 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:08.967447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:08.967591 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:08.968017 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:08.968023 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:08.968026 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:08.968414 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:08.968761 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:08.969096 (+ 335us) server_negotiation.cc:299] Negotiation successful
0806 02:59:08.969397 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":348,"threads_started":1}
Aug 06 02:59:08 dist-test-slave-dhph krb5kdc[3241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 02:59:08.996873 3759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:08.977152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37392 (local address 127.31.74.254:38781)
0806 02:59:08.977499 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:08.977506 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:08.977759 (+ 253us) server_negotiation.cc:407] Connection header received
0806 02:59:08.978000 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:08.978011 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:08.978102 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:08.978225 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:08.980047 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:08.980789 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:08.982038 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:08.982764 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:08.982928 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:08.988157 (+ 5229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:08.988163 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:08.988166 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:08.988208 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:08.992386 (+ 4178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:08.993977 (+ 1591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:08.993982 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:08.993985 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:08.994141 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:08.994780 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:08.994788 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:08.994791 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:08.995396 (+ 605us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:08.995728 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:08.996213 (+ 485us) server_negotiation.cc:299] Negotiation successful
0806 02:59:08.996535 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153}
W20260806 02:59:08.997593 3287 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37392
I20260806 02:59:08.999177 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 3351
I20260806 02:59:09.019465 3765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:09.000985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44200 (local address 127.31.74.193:42163)
0806 02:59:09.001233 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:09.001238 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:09.001427 (+ 189us) server_negotiation.cc:407] Connection header received
0806 02:59:09.001716 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:09.001721 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:09.001881 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:09.001976 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:09.003679 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:09.004812 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:09.007515 (+ 2703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:09.008025 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:09.008400 (+ 375us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:09.012170 (+ 3770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:09.012176 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:09.012183 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:09.012227 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:09.015756 (+ 3529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:09.016730 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:09.016735 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:09.016738 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:09.016901 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:09.017518 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:09.017523 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:09.017525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:09.017883 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:09.018096 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:09.018797 (+ 701us) server_negotiation.cc:299] Negotiation successful
0806 02:59:09.019063 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260806 02:59:09.020263 3478 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:59:09.124725 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 3351
I20260806 02:59:09.153330 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 3489
Aug 06 02:59:09 dist-test-slave-dhph krb5kdc[3241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.74.194@KRBTEST.COM
I20260806 02:59:09.180929 3770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:09.154821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48840 (local address 127.31.74.194:36201)
0806 02:59:09.155583 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:09.155592 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:09.155619 (+ 27us) server_negotiation.cc:407] Connection header received
0806 02:59:09.155694 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:09.155701 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:09.155936 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:09.156076 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:09.160258 (+ 4182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:09.161558 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:09.163186 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:09.164233 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:09.164409 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:09.172512 (+ 8103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:09.172542 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:09.172566 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:09.172646 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:09.177124 (+ 4478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:09.178126 (+ 1002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:09.178135 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:09.178138 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:09.178320 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:09.178904 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:09.178908 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:09.178911 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:09.179383 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:09.179770 (+ 387us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:09.180151 (+ 381us) server_negotiation.cc:299] Negotiation successful
0806 02:59:09.180495 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":315,"threads_started":1}
I20260806 02:59:09.183056 3614 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:59:09.285610 3619 heartbeater.cc:503] Master 127.31.74.254:38781 was elected leader, sending a full tablet report...
I20260806 02:59:09.287905 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 3489
W20260806 02:59:09.312781 3270 connection.cc:535] server connection from 127.31.74.194:39363 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260806 02:59:09.313290 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 3626
Aug 06 02:59:09 dist-test-slave-dhph krb5kdc[3241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.74.195@KRBTEST.COM
I20260806 02:59:09.336759 3772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:09.314656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52126 (local address 127.31.74.195:42821)
0806 02:59:09.315457 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:09.315465 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:09.315502 (+ 37us) server_negotiation.cc:407] Connection header received
0806 02:59:09.315573 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:09.315580 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:09.315812 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:09.315986 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:09.317705 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:09.319038 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:09.320519 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:09.321100 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:09.321268 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:09.327659 (+ 6391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:09.327691 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:09.327717 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:09.327776 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:09.332609 (+ 4833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:09.333879 (+ 1270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:09.333887 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:09.333890 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:09.334053 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:09.334567 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:09.334572 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:09.334574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:09.335191 (+ 617us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:09.335451 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:09.336061 (+ 610us) server_negotiation.cc:299] Negotiation successful
0806 02:59:09.336293 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":316,"threads_started":1}
I20260806 02:59:09.338568 3751 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:59:09.453485 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 3626
I20260806 02:59:09.480957 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 3254
I20260806 02:59:09.482460 3319 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:59:09.612008 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 3254
2026-08-06T02:59:09Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (5438 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:59:09 dist-test-slave-dhph krb5kdc[3777](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:59:09 dist-test-slave-dhph krb5kdc[3777](info): set up 2 sockets
Aug 06 02:59:09 dist-test-slave-dhph krb5kdc[3777](info): commencing operation
krb5kdc: starting...
W20260806 02:59:10.955672 32043 mini_kdc.cc:121] Time spent starting KDC: real 1.289s user 0.004s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/krb5kdc/test-user.keytab.
Aug 06 02:59:11 dist-test-slave-dhph krb5kdc[3777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985151, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:59:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:59:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.74.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab.
Entry for principal kudu/127.31.74.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab.
I20260806 02:59:11.286487 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:42793
--webserver_interface=127.31.74.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab
--principal=kudu/127.31.74.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:44945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.74.254:42793
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:59:11.797441 3790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:59:11.797942 3790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:59:11.798143 3790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:59:11.808499 3790 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:59:11.808807 3790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:59:11.809121 3790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:59:11.809304 3790 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:59:11.809407 3790 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:59:11.822335 3790 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:44945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.74.254:42793
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab
--principal=kudu/127.31.74.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:42793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.74.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:59:11.823979 3790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:59:11.826128 3790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:59:11.851217 3797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:11.851011 3796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:11.851715 3799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:59:11.858454 3790 server_base.cc:1034] running on GCE node
I20260806 02:59:11.859726 3790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:59:11.861577 3790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:59:11.862886 3790 hybrid_clock.cc:648] HybridClock initialized: now 1785985151862836 us; error 81 us; skew 500 ppm
I20260806 02:59:11.863330 3790 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:59:11 dist-test-slave-dhph krb5kdc[3777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985151, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:59:11.869320 3790 init.cc:377] Logged in from keytab as kudu/127.31.74.254@KRBTEST.COM (short username kudu)
I20260806 02:59:11.872716 3790 fs_manager.cc:359] Metadata directory not provided
I20260806 02:59:11.873004 3790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:59:11.873817 3790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:59:11.879796 3790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/minicluster-data/master-0/data/instance:
uuid: "dd161421a51e44f2a55eb59929352a07"
format_stamp: "Formatted at 2026-08-06 02:59:11 on dist-test-slave-dhph"
server_key: "4b7072cccbe78d0f69ad91db094058c4"
server_key_iv: "6dedb45c39eae17f387303061127f63c"
server_key_version: "encryptionkey@0"
I20260806 02:59:11.881114 3790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/minicluster-data/master-0/wal/instance:
uuid: "dd161421a51e44f2a55eb59929352a07"
format_stamp: "Formatted at 2026-08-06 02:59:11 on dist-test-slave-dhph"
server_key: "4b7072cccbe78d0f69ad91db094058c4"
server_key_iv: "6dedb45c39eae17f387303061127f63c"
server_key_version: "encryptionkey@0"
I20260806 02:59:11.888978 3790 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20260806 02:59:11.894122 3805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:11.896346 3790 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260806 02:59:11.896639 3790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/minicluster-data/master-0/wal
uuid: "dd161421a51e44f2a55eb59929352a07"
format_stamp: "Formatted at 2026-08-06 02:59:11 on dist-test-slave-dhph"
server_key: "4b7072cccbe78d0f69ad91db094058c4"
server_key_iv: "6dedb45c39eae17f387303061127f63c"
server_key_version: "encryptionkey@0"
I20260806 02:59:11.896974 3790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:59:11.953792 3790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:59:11.961365 3790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:59:11.961741 3790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:59:11.992535 3856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.254:42793 every 8 connection(s)
I20260806 02:59:11.992857 3790 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.254:42793
I20260806 02:59:11.993947 3790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
I20260806 02:59:11.998875 3857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:59:11.999100 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 3790
I20260806 02:59:11.999377 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/minicluster-data/master-0/wal/instance
I20260806 02:59:12.000568 32043 external_mini_cluster.cc:1428] Setting key 615a58e6e1cda7254387bbf1236a72ee
I20260806 02:59:12.024026 3857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd161421a51e44f2a55eb59929352a07: Bootstrap starting.
Aug 06 02:59:12 dist-test-slave-dhph krb5kdc[3777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985151, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 02:59:12.029230 3857 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd161421a51e44f2a55eb59929352a07: Neither blocks nor log segments found. Creating new log.
I20260806 02:59:12.030923 3857 log.cc:824] T 00000000000000000000000000000000 P dd161421a51e44f2a55eb59929352a07: Log is configured to *not* fsync() on all Append() calls
I20260806 02:59:12.039213 3857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd161421a51e44f2a55eb59929352a07: No bootstrap required, opened a new log
I20260806 02:59:12.040892 3860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:12.004290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58598 (local address 127.31.74.254:42793)
0806 02:59:12.006073 (+ 1783us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:12.006086 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:12.006139 (+ 53us) server_negotiation.cc:407] Connection header received
0806 02:59:12.008289 (+ 2150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:12.008321 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:12.008871 (+ 550us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:12.009332 (+ 461us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:12.012030 (+ 2698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:12.014017 (+ 1987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:12.018066 (+ 4049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:12.019567 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:12.020365 (+ 798us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:12.028962 (+ 8597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:12.028976 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:12.029003 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:12.029094 (+ 91us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:12.033622 (+ 4528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:12.034800 (+ 1178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:12.034809 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:12.034818 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:12.035046 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:12.035542 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:12.035546 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:12.035549 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:12.038100 (+ 2551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:12.038479 (+ 379us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:12.039136 (+ 657us) server_negotiation.cc:299] Negotiation successful
0806 02:59:12.039657 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1502,"thread_start_us":875,"threads_started":1}
I20260806 02:59:12.049647 3857 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd161421a51e44f2a55eb59929352a07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd161421a51e44f2a55eb59929352a07" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 42793 } }
I20260806 02:59:12.050144 3857 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd161421a51e44f2a55eb59929352a07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:59:12.050268 3857 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd161421a51e44f2a55eb59929352a07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd161421a51e44f2a55eb59929352a07, State: Initialized, Role: FOLLOWER
I20260806 02:59:12.051200 3857 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd161421a51e44f2a55eb59929352a07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd161421a51e44f2a55eb59929352a07" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 42793 } }
I20260806 02:59:12.051513 3857 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd161421a51e44f2a55eb59929352a07 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:59:12.051672 3857 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd161421a51e44f2a55eb59929352a07 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:59:12.051873 3857 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd161421a51e44f2a55eb59929352a07 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:59:12.054296 3857 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd161421a51e44f2a55eb59929352a07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd161421a51e44f2a55eb59929352a07" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 42793 } }
I20260806 02:59:12.055066 3857 leader_election.cc:302] T 00000000000000000000000000000000 P dd161421a51e44f2a55eb59929352a07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd161421a51e44f2a55eb59929352a07; no voters:
I20260806 02:59:12.055801 3857 leader_election.cc:288] T 00000000000000000000000000000000 P dd161421a51e44f2a55eb59929352a07 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:59:12.056100 3862 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd161421a51e44f2a55eb59929352a07 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:59:12.057559 3862 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd161421a51e44f2a55eb59929352a07 [term 1 LEADER]: Becoming Leader. State: Replica: dd161421a51e44f2a55eb59929352a07, State: Running, Role: LEADER
I20260806 02:59:12.058231 3862 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd161421a51e44f2a55eb59929352a07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd161421a51e44f2a55eb59929352a07" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 42793 } }
I20260806 02:59:12.058427 3857 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd161421a51e44f2a55eb59929352a07 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:59:12.061932 3863 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd161421a51e44f2a55eb59929352a07 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd161421a51e44f2a55eb59929352a07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd161421a51e44f2a55eb59929352a07" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 42793 } } }
I20260806 02:59:12.061908 3864 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd161421a51e44f2a55eb59929352a07 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd161421a51e44f2a55eb59929352a07. Latest consensus state: current_term: 1 leader_uuid: "dd161421a51e44f2a55eb59929352a07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd161421a51e44f2a55eb59929352a07" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 42793 } } }
I20260806 02:59:12.062502 3863 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd161421a51e44f2a55eb59929352a07 [sys.catalog]: This master's current role is: LEADER
I20260806 02:59:12.062500 3864 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd161421a51e44f2a55eb59929352a07 [sys.catalog]: This master's current role is: LEADER
I20260806 02:59:12.063742 3871 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:59:12.074155 3871 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:59:12.088275 3871 catalog_manager.cc:1347] Generated new cluster ID: afa7ad57f5a54799a7c43f9de946271f
I20260806 02:59:12.088440 3871 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:59:12.098793 3871 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:59:12.101055 3871 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:59:12.117596 3871 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd161421a51e44f2a55eb59929352a07: Generated new TSK 0
I20260806 02:59:12.119095 3871 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.74.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.193.keytab.
Entry for principal kudu/127.31.74.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.193.keytab.
I20260806 02:59:12.220216 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.193:0
--local_ip_for_outbound_sockets=127.31.74.193
--webserver_interface=127.31.74.193
--webserver_port=0
--tserver_master_addrs=127.31.74.254:42793
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.193.keytab
--principal=kudu/127.31.74.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:44945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:59:12.679029 3883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:59:12.679764 3883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:59:12.680055 3883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:59:12.689639 3883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:59:12.689791 3883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:59:12.690023 3883 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.193
I20260806 02:59:12.702672 3883 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:44945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.193.keytab
--principal=kudu/127.31.74.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.74.193
--webserver_port=0
--tserver_master_addrs=127.31.74.254:42793
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.193
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:59:12.704332 3883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:59:12.705826 3883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:59:12.723903 3889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:12.723990 3892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:12.723958 3890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:59:12.725996 3883 server_base.cc:1034] running on GCE node
I20260806 02:59:12.727355 3883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:59:12.729208 3883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:59:12.730557 3883 hybrid_clock.cc:648] HybridClock initialized: now 1785985152730487 us; error 66 us; skew 500 ppm
I20260806 02:59:12.731153 3883 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:59:12 dist-test-slave-dhph krb5kdc[3777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985152, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:59:12.735085 3883 init.cc:377] Logged in from keytab as kudu/127.31.74.193@KRBTEST.COM (short username kudu)
I20260806 02:59:12.736567 3883 fs_manager.cc:359] Metadata directory not provided
I20260806 02:59:12.736717 3883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:59:12.737058 3883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:59:12.740865 3883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/minicluster-data/ts-0/data/instance:
uuid: "f1d23fb9d05d49debe91b1df45f52169"
format_stamp: "Formatted at 2026-08-06 02:59:12 on dist-test-slave-dhph"
server_key: "638fa88831a3a321564d006c2fae5b4c"
server_key_iv: "59c980ba52ff068030090d9506b17d74"
server_key_version: "encryptionkey@0"
I20260806 02:59:12.742723 3883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/minicluster-data/ts-0/wal/instance:
uuid: "f1d23fb9d05d49debe91b1df45f52169"
format_stamp: "Formatted at 2026-08-06 02:59:12 on dist-test-slave-dhph"
server_key: "638fa88831a3a321564d006c2fae5b4c"
server_key_iv: "59c980ba52ff068030090d9506b17d74"
server_key_version: "encryptionkey@0"
I20260806 02:59:12.751698 3883 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.002s sys 0.005s
I20260806 02:59:12.756738 3899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:12.758946 3883 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20260806 02:59:12.759173 3883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/minicluster-data/ts-0/wal
uuid: "f1d23fb9d05d49debe91b1df45f52169"
format_stamp: "Formatted at 2026-08-06 02:59:12 on dist-test-slave-dhph"
server_key: "638fa88831a3a321564d006c2fae5b4c"
server_key_iv: "59c980ba52ff068030090d9506b17d74"
server_key_version: "encryptionkey@0"
I20260806 02:59:12.759464 3883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:59:12.783178 3883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:59:12.789115 3883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:59:12.789446 3883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:59:12.790936 3883 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:59:12.792457 3883 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:59:12.792582 3883 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:12.792734 3883 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:59:12.792835 3883 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:12.839167 3883 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.193:38641
I20260806 02:59:12.839237 4012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.193:38641 every 8 connection(s)
I20260806 02:59:12.840312 3883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
I20260806 02:59:12.842609 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 3883
I20260806 02:59:12.842888 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/minicluster-data/ts-0/wal/instance
I20260806 02:59:12.844038 32043 external_mini_cluster.cc:1428] Setting key 49a582a21b89890b7c672a4605847166
Aug 06 02:59:12 dist-test-slave-dhph krb5kdc[3777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985152, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.193@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 02:59:12.869122 4015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:12.843648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.193:39205 (local address 127.31.74.254:42793)
0806 02:59:12.844340 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:12.844349 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:12.845560 (+ 1211us) server_negotiation.cc:407] Connection header received
0806 02:59:12.848703 (+ 3143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:12.848708 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:12.848806 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:12.848945 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:12.850669 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:12.851597 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:12.853462 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:12.853920 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:12.854077 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:12.859422 (+ 5345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:12.859429 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:12.859433 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:12.859486 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:12.862940 (+ 3454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:12.865610 (+ 2670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:12.865617 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:12.865620 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:12.865901 (+ 281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:12.866754 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:12.866761 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:12.866764 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:12.867186 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:12.867374 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:12.868565 (+ 1191us) server_negotiation.cc:299] Negotiation successful
0806 02:59:12.868831 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":309,"threads_started":1}
I20260806 02:59:12.871690 4016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:12.844468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:42793 (local address 127.31.74.193:39205)
0806 02:59:12.845387 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:12.845431 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:12.848374 (+ 2943us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:12.849210 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:12.849220 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:12.849815 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:59:12.850437 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:12.850455 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:12.851809 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:12.851814 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:12.853288 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:12.853302 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:12.854212 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:12.854220 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:12.854473 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:12.855386 (+ 913us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:59:12.855418 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:59:12.858944 (+ 3526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:59:12.863477 (+ 4533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:12.863499 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:12.863534 (+ 35us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:12.865029 (+ 1495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:12.866116 (+ 1087us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:12.866125 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:12.866133 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:12.866502 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:12.867379 (+ 877us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:59:12.867393 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:59:12.868018 (+ 625us) client_negotiation.cc:769] Sending connection context
0806 02:59:12.868426 (+ 408us) client_negotiation.cc:240] Negotiation successful
0806 02:59:12.869209 (+ 783us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"thread_start_us":408,"threads_started":1}
WARNING: no policy specified for kudu/127.31.74.194@KRBTEST.COM; defaulting to no policy
I20260806 02:59:12.874562 4013 heartbeater.cc:347] Connected to a master server at 127.31.74.254:42793
I20260806 02:59:12.875124 4013 heartbeater.cc:464] Registering TS with master...
I20260806 02:59:12.876259 4013 heartbeater.cc:511] Master 127.31.74.254:42793 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.194@KRBTEST.COM" created.
I20260806 02:59:12.879736 3822 ts_manager.cc:194] Registered new tserver with Master: f1d23fb9d05d49debe91b1df45f52169 (127.31.74.193:38641)
I20260806 02:59:12.881481 3822 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.74.193@KRBTEST.COM'} at 127.31.74.193:39205
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.194.keytab.
Entry for principal kudu/127.31.74.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.194.keytab.
I20260806 02:59:12.932762 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.194:0
--local_ip_for_outbound_sockets=127.31.74.194
--webserver_interface=127.31.74.194
--webserver_port=0
--tserver_master_addrs=127.31.74.254:42793
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.194.keytab
--principal=kudu/127.31.74.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:44945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:59:13.379384 4019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:59:13.379719 4019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:59:13.379892 4019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:59:13.389220 4019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:59:13.389669 4019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:59:13.389967 4019 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.194
I20260806 02:59:13.405504 4019 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:44945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.194.keytab
--principal=kudu/127.31.74.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.74.194
--webserver_port=0
--tserver_master_addrs=127.31.74.254:42793
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.194
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:59:13.407276 4019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:59:13.409191 4019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:59:13.430212 4024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:13.430186 4025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:13.430441 4027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:59:13.433040 4019 server_base.cc:1034] running on GCE node
I20260806 02:59:13.433849 4019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:59:13.435501 4019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:59:13.436817 4019 hybrid_clock.cc:648] HybridClock initialized: now 1785985153436729 us; error 74 us; skew 500 ppm
I20260806 02:59:13.437275 4019 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:59:13 dist-test-slave-dhph krb5kdc[3777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985153, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:59:13.442080 4019 init.cc:377] Logged in from keytab as kudu/127.31.74.194@KRBTEST.COM (short username kudu)
I20260806 02:59:13.443718 4019 fs_manager.cc:359] Metadata directory not provided
I20260806 02:59:13.443886 4019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:59:13.444279 4019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:59:13.448959 4019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/minicluster-data/ts-1/data/instance:
uuid: "a7137cef4cab448bb0d50e90c1342c5f"
format_stamp: "Formatted at 2026-08-06 02:59:13 on dist-test-slave-dhph"
server_key: "f0e3ddb034de52bf45ad12065ef61125"
server_key_iv: "6dba560704bf06b0b5fdd7a53774d52b"
server_key_version: "encryptionkey@0"
I20260806 02:59:13.450316 4019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/minicluster-data/ts-1/wal/instance:
uuid: "a7137cef4cab448bb0d50e90c1342c5f"
format_stamp: "Formatted at 2026-08-06 02:59:13 on dist-test-slave-dhph"
server_key: "f0e3ddb034de52bf45ad12065ef61125"
server_key_iv: "6dba560704bf06b0b5fdd7a53774d52b"
server_key_version: "encryptionkey@0"
I20260806 02:59:13.457571 4019 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.003s
I20260806 02:59:13.463289 4033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:13.465481 4019 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.000s sys 0.004s
I20260806 02:59:13.465709 4019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/minicluster-data/ts-1/wal
uuid: "a7137cef4cab448bb0d50e90c1342c5f"
format_stamp: "Formatted at 2026-08-06 02:59:13 on dist-test-slave-dhph"
server_key: "f0e3ddb034de52bf45ad12065ef61125"
server_key_iv: "6dba560704bf06b0b5fdd7a53774d52b"
server_key_version: "encryptionkey@0"
I20260806 02:59:13.465992 4019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:59:13.497887 4019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:59:13.503369 4019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:59:13.503696 4019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:59:13.505141 4019 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:59:13.507261 4019 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:59:13.507380 4019 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:13.507799 4019 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:59:13.508222 4019 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:13.552708 4019 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.194:40727
I20260806 02:59:13.552865 4150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.194:40727 every 8 connection(s)
I20260806 02:59:13.553705 4019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
I20260806 02:59:13.558888 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 4019
I20260806 02:59:13.559121 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/minicluster-data/ts-1/wal/instance
I20260806 02:59:13.559723 32043 external_mini_cluster.cc:1428] Setting key dac9f79a1ef478956f87382c74dc3b0f
Aug 06 02:59:13 dist-test-slave-dhph krb5kdc[3777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985153, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.194@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 02:59:13.581250 4153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:13.557056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.194:50263 (local address 127.31.74.254:42793)
0806 02:59:13.557596 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:13.557602 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:13.558911 (+ 1309us) server_negotiation.cc:407] Connection header received
0806 02:59:13.561194 (+ 2283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:13.561200 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:13.561296 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:13.561433 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:13.563280 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:13.564367 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:13.566358 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:13.567084 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:13.567295 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:13.572011 (+ 4716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:13.572017 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:13.572021 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:13.572064 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:13.576556 (+ 4492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:13.577663 (+ 1107us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:13.577673 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:13.577680 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:13.577878 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:13.578415 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:13.578424 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:13.578431 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:13.578933 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:13.579105 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:13.580336 (+ 1231us) server_negotiation.cc:299] Negotiation successful
0806 02:59:13.580692 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":286,"threads_started":1}
I20260806 02:59:13.581470 4154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:13.557717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:42793 (local address 127.31.74.194:50263)
0806 02:59:13.558727 (+ 1010us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:13.558778 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:13.560893 (+ 2115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:13.561708 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:13.561717 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:13.562326 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:59:13.563068 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:13.563088 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:13.564545 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:13.564552 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:13.566175 (+ 1623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:13.566189 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:13.567416 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:13.567424 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:13.567694 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:13.568477 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:59:13.568502 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:59:13.571701 (+ 3199us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:59:13.576822 (+ 5121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:13.576833 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:13.576852 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:13.577475 (+ 623us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:13.578039 (+ 564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:13.578043 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:13.578046 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:13.578268 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:13.579100 (+ 832us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:59:13.579108 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:59:13.579513 (+ 405us) client_negotiation.cc:769] Sending connection context
0806 02:59:13.579846 (+ 333us) client_negotiation.cc:240] Negotiation successful
0806 02:59:13.580305 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":652,"thread_start_us":338,"threads_started":1}
I20260806 02:59:13.583246 4151 heartbeater.cc:347] Connected to a master server at 127.31.74.254:42793
I20260806 02:59:13.583613 4151 heartbeater.cc:464] Registering TS with master...
I20260806 02:59:13.584683 4151 heartbeater.cc:511] Master 127.31.74.254:42793 requested a full tablet report, sending...
I20260806 02:59:13.586882 3822 ts_manager.cc:194] Registered new tserver with Master: a7137cef4cab448bb0d50e90c1342c5f (127.31.74.194:40727)
I20260806 02:59:13.587872 3822 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.74.194@KRBTEST.COM'} at 127.31.74.194:50263
WARNING: no policy specified for kudu/127.31.74.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.195.keytab.
Entry for principal kudu/127.31.74.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.195.keytab.
I20260806 02:59:13.645195 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.195:0
--local_ip_for_outbound_sockets=127.31.74.195
--webserver_interface=127.31.74.195
--webserver_port=0
--tserver_master_addrs=127.31.74.254:42793
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.195.keytab
--principal=kudu/127.31.74.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:44945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:59:13.885792 4013 heartbeater.cc:503] Master 127.31.74.254:42793 was elected leader, sending a full tablet report...
W20260806 02:59:14.058722 4157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:59:14.059140 4157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:59:14.059350 4157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:59:14.068470 4157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:59:14.068614 4157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:59:14.068825 4157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.195
I20260806 02:59:14.081727 4157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:44945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.195.keytab
--principal=kudu/127.31.74.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.74.195
--webserver_port=0
--tserver_master_addrs=127.31.74.254:42793
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.195
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:59:14.083670 4157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:59:14.085219 4157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:59:14.102622 4163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:14.102965 4162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:14.103081 4165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:59:14.104914 4157 server_base.cc:1034] running on GCE node
I20260806 02:59:14.105705 4157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:59:14.106850 4157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:59:14.108134 4157 hybrid_clock.cc:648] HybridClock initialized: now 1785985154108036 us; error 84 us; skew 500 ppm
I20260806 02:59:14.108516 4157 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:59:14 dist-test-slave-dhph krb5kdc[3777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985154, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:59:14.112723 4157 init.cc:377] Logged in from keytab as kudu/127.31.74.195@KRBTEST.COM (short username kudu)
I20260806 02:59:14.114429 4157 fs_manager.cc:359] Metadata directory not provided
I20260806 02:59:14.114554 4157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:59:14.114986 4157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:59:14.118572 4157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/minicluster-data/ts-2/data/instance:
uuid: "6647d228143a4c3987f8b33924a6f317"
format_stamp: "Formatted at 2026-08-06 02:59:14 on dist-test-slave-dhph"
server_key: "336fbae6454c482fa95094172799b49a"
server_key_iv: "614becdbd6b0dfac02091b788a0ba3f4"
server_key_version: "encryptionkey@0"
I20260806 02:59:14.119639 4157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/minicluster-data/ts-2/wal/instance:
uuid: "6647d228143a4c3987f8b33924a6f317"
format_stamp: "Formatted at 2026-08-06 02:59:14 on dist-test-slave-dhph"
server_key: "336fbae6454c482fa95094172799b49a"
server_key_iv: "614becdbd6b0dfac02091b788a0ba3f4"
server_key_version: "encryptionkey@0"
I20260806 02:59:14.127063 4157 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.005s
I20260806 02:59:14.132426 4171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:14.134545 4157 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.003s
I20260806 02:59:14.134897 4157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/minicluster-data/ts-2/wal
uuid: "6647d228143a4c3987f8b33924a6f317"
format_stamp: "Formatted at 2026-08-06 02:59:14 on dist-test-slave-dhph"
server_key: "336fbae6454c482fa95094172799b49a"
server_key_iv: "614becdbd6b0dfac02091b788a0ba3f4"
server_key_version: "encryptionkey@0"
I20260806 02:59:14.135192 4157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:59:14.164510 4157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:59:14.171326 4157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:59:14.171667 4157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:59:14.172871 4157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:59:14.174420 4157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:59:14.174508 4157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:14.174687 4157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:59:14.174768 4157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:14.217476 4157 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.195:45757
I20260806 02:59:14.217574 4285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.195:45757 every 8 connection(s)
I20260806 02:59:14.218429 4157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
I20260806 02:59:14.222641 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 4157
I20260806 02:59:14.222886 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/minicluster-data/ts-2/wal/instance
I20260806 02:59:14.223390 32043 external_mini_cluster.cc:1428] Setting key 194590cc6f666205837abe3d0db39eb0
Aug 06 02:59:14 dist-test-slave-dhph krb5kdc[3777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985154, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.195@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 02:59:14.245452 4289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:14.222306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.195:45639 (local address 127.31.74.254:42793)
0806 02:59:14.222958 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:14.222967 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:14.224060 (+ 1093us) server_negotiation.cc:407] Connection header received
0806 02:59:14.226019 (+ 1959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:14.226024 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:14.226144 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:14.226266 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:14.228254 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:14.229294 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:14.231515 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:14.232011 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:14.232177 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:14.236648 (+ 4471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:14.236653 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:14.236657 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:14.236701 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:14.240827 (+ 4126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:14.241724 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:14.241735 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:14.241739 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:14.242083 (+ 344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:14.243057 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:14.243063 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:14.243068 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:14.243547 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:14.243766 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:14.244598 (+ 832us) server_negotiation.cc:299] Negotiation successful
0806 02:59:14.244878 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":322,"threads_started":1}
I20260806 02:59:14.246529 4290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:14.222980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:42793 (local address 127.31.74.195:45639)
0806 02:59:14.223852 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:14.223910 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:14.225723 (+ 1813us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:14.226541 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:14.226549 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:14.227355 (+ 806us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:59:14.228044 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:14.228065 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:14.229505 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:14.229516 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:14.231319 (+ 1803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:14.231336 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:14.232312 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:14.232321 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:14.232626 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:14.233509 (+ 883us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:59:14.233542 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:59:14.236314 (+ 2772us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:59:14.241001 (+ 4687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:14.241012 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:14.241033 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:14.241571 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:14.242571 (+ 1000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:14.242608 (+ 37us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:14.242612 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:14.242905 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:14.243725 (+ 820us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:59:14.243735 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:59:14.244158 (+ 423us) client_negotiation.cc:769] Sending connection context
0806 02:59:14.244490 (+ 332us) client_negotiation.cc:240] Negotiation successful
0806 02:59:14.245119 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":362,"threads_started":1}
I20260806 02:59:14.248394 4286 heartbeater.cc:347] Connected to a master server at 127.31.74.254:42793
I20260806 02:59:14.248915 4286 heartbeater.cc:464] Registering TS with master...
I20260806 02:59:14.249747 4286 heartbeater.cc:511] Master 127.31.74.254:42793 requested a full tablet report, sending...
I20260806 02:59:14.251914 3822 ts_manager.cc:194] Registered new tserver with Master: 6647d228143a4c3987f8b33924a6f317 (127.31.74.195:45757)
I20260806 02:59:14.252876 3822 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.74.195@KRBTEST.COM'} at 127.31.74.195:45639
I20260806 02:59:14.254653 32043 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:59:14.278285 4289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:14.261711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58606 (local address 127.31.74.254:42793)
0806 02:59:14.261938 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:14.261944 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:14.262630 (+ 686us) server_negotiation.cc:407] Connection header received
0806 02:59:14.262906 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:14.262911 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:14.263023 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:14.263161 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:14.265143 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:14.265978 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:14.267509 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:14.267988 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:14.268148 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:14.271263 (+ 3115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:14.271268 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:14.271272 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:14.271311 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:14.275098 (+ 3787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:14.275949 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:14.275954 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:14.275957 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:14.276094 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:14.276583 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:14.276587 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:14.276589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:14.276974 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:14.277260 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:14.277637 (+ 377us) server_negotiation.cc:299] Negotiation successful
0806 02:59:14.277920 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260806 02:59:14.283574 3822 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58606:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:59:14.291420 3822 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:59:14.325706 4302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:14.309376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.195:45757 (local address 127.0.0.1:60522)
0806 02:59:14.311491 (+ 2115us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:14.311541 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:14.311666 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:14.313545 (+ 1879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:14.313558 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:14.313604 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:59:14.314128 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:14.314150 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:14.317646 (+ 3496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:14.317656 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:14.320832 (+ 3176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:14.320855 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:14.323743 (+ 2888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:14.323751 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:14.324015 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:14.324089 (+ 74us) client_negotiation.cc:769] Sending connection context
0806 02:59:14.324259 (+ 170us) client_negotiation.cc:240] Negotiation successful
0806 02:59:14.324607 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1993,"mutex_wait_us":56,"thread_start_us":1741,"threads_started":1}
I20260806 02:59:14.328440 4303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:14.309851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60522 (local address 127.31.74.195:45757)
0806 02:59:14.312622 (+ 2771us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:14.312637 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:14.312687 (+ 50us) server_negotiation.cc:407] Connection header received
0806 02:59:14.312815 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:14.312824 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:14.313187 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:14.314520 (+ 1333us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:59:14.315025 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:14.317261 (+ 2236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:14.321153 (+ 3892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:14.323457 (+ 2304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:14.326996 (+ 3539us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:14.327638 (+ 642us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:14.327764 (+ 126us) server_negotiation.cc:299] Negotiation successful
0806 02:59:14.328042 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2547,"thread_start_us":2341,"threads_started":1}
I20260806 02:59:14.335467 4304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:14.309949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59738 (local address 127.31.74.194:40727)
0806 02:59:14.325497 (+ 15548us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:14.325509 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:14.325565 (+ 56us) server_negotiation.cc:407] Connection header received
0806 02:59:14.325749 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:14.325764 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:14.326273 (+ 509us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:14.328088 (+ 1815us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:59:14.328400 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:14.330227 (+ 1827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:14.333071 (+ 2844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:14.334175 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:14.334364 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:14.334665 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:14.334916 (+ 251us) server_negotiation.cc:299] Negotiation successful
0806 02:59:14.335095 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15194,"thread_start_us":16693,"threads_started":1}
I20260806 02:59:14.335533 4301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:14.309365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.194:40727 (local address 127.0.0.1:59738)
0806 02:59:14.310441 (+ 1076us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:14.310505 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:14.310787 (+ 282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:14.327300 (+ 16513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:14.327307 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:14.327350 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:59:14.327799 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:14.327815 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:14.330424 (+ 2609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:14.330434 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:14.332820 (+ 2386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:14.332844 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:14.334423 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:14.334431 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:14.334674 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:14.334727 (+ 53us) client_negotiation.cc:769] Sending connection context
0806 02:59:14.334849 (+ 122us) client_negotiation.cc:240] Negotiation successful
0806 02:59:14.335014 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":683,"mutex_wait_us":62,"thread_start_us":777,"threads_started":1}
I20260806 02:59:14.336192 4221 tablet_service.cc:1467] Processing CreateTablet for tablet 1493ae7d57cc424092cb0bc35aa65ff0 (DEFAULT_TABLE table=test-table [id=1978fe62e881430aab4d8e993e60ae43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:59:14.337800 4302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:14.312425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:38641 (local address 127.0.0.1:48996)
0806 02:59:14.326320 (+ 13895us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:14.326375 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:14.326637 (+ 262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:14.328259 (+ 1622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:14.328266 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:14.328296 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:59:14.328544 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:14.328557 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:14.331016 (+ 2459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:14.331023 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:14.333071 (+ 2048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:14.333087 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:14.336864 (+ 3777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:14.336872 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:14.337082 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:14.337132 (+ 50us) client_negotiation.cc:769] Sending connection context
0806 02:59:14.337246 (+ 114us) client_negotiation.cc:240] Negotiation successful
0806 02:59:14.337417 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13723,"mutex_wait_us":47,"thread_start_us":18201,"threads_started":1}
I20260806 02:59:14.337987 4306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:14.312757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48996 (local address 127.31.74.193:38641)
0806 02:59:14.327036 (+ 14279us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:14.327049 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:14.327098 (+ 49us) server_negotiation.cc:407] Connection header received
0806 02:59:14.327297 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:14.327308 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:14.327695 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:14.328864 (+ 1169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:59:14.329113 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:14.330828 (+ 1715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:14.333269 (+ 2441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:14.336621 (+ 3352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:14.336801 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:14.337083 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:14.337419 (+ 336us) server_negotiation.cc:299] Negotiation successful
0806 02:59:14.337633 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13893,"thread_start_us":15159,"threads_started":1}
I20260806 02:59:14.338034 4221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1493ae7d57cc424092cb0bc35aa65ff0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:59:14.341223 4084 tablet_service.cc:1467] Processing CreateTablet for tablet 1493ae7d57cc424092cb0bc35aa65ff0 (DEFAULT_TABLE table=test-table [id=1978fe62e881430aab4d8e993e60ae43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:59:14.343029 4084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1493ae7d57cc424092cb0bc35aa65ff0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:59:14.343624 3947 tablet_service.cc:1467] Processing CreateTablet for tablet 1493ae7d57cc424092cb0bc35aa65ff0 (DEFAULT_TABLE table=test-table [id=1978fe62e881430aab4d8e993e60ae43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:59:14.345075 3947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1493ae7d57cc424092cb0bc35aa65ff0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:59:14.360426 4307 tablet_bootstrap.cc:492] T 1493ae7d57cc424092cb0bc35aa65ff0 P f1d23fb9d05d49debe91b1df45f52169: Bootstrap starting.
I20260806 02:59:14.363210 4308 tablet_bootstrap.cc:492] T 1493ae7d57cc424092cb0bc35aa65ff0 P 6647d228143a4c3987f8b33924a6f317: Bootstrap starting.
I20260806 02:59:14.365942 4309 tablet_bootstrap.cc:492] T 1493ae7d57cc424092cb0bc35aa65ff0 P a7137cef4cab448bb0d50e90c1342c5f: Bootstrap starting.
I20260806 02:59:14.367949 4307 tablet_bootstrap.cc:654] T 1493ae7d57cc424092cb0bc35aa65ff0 P f1d23fb9d05d49debe91b1df45f52169: Neither blocks nor log segments found. Creating new log.
I20260806 02:59:14.368168 4308 tablet_bootstrap.cc:654] T 1493ae7d57cc424092cb0bc35aa65ff0 P 6647d228143a4c3987f8b33924a6f317: Neither blocks nor log segments found. Creating new log.
I20260806 02:59:14.369488 4309 tablet_bootstrap.cc:654] T 1493ae7d57cc424092cb0bc35aa65ff0 P a7137cef4cab448bb0d50e90c1342c5f: Neither blocks nor log segments found. Creating new log.
I20260806 02:59:14.369580 4307 log.cc:824] T 1493ae7d57cc424092cb0bc35aa65ff0 P f1d23fb9d05d49debe91b1df45f52169: Log is configured to *not* fsync() on all Append() calls
I20260806 02:59:14.369582 4308 log.cc:824] T 1493ae7d57cc424092cb0bc35aa65ff0 P 6647d228143a4c3987f8b33924a6f317: Log is configured to *not* fsync() on all Append() calls
I20260806 02:59:14.370963 4309 log.cc:824] T 1493ae7d57cc424092cb0bc35aa65ff0 P a7137cef4cab448bb0d50e90c1342c5f: Log is configured to *not* fsync() on all Append() calls
I20260806 02:59:14.373947 4308 tablet_bootstrap.cc:492] T 1493ae7d57cc424092cb0bc35aa65ff0 P 6647d228143a4c3987f8b33924a6f317: No bootstrap required, opened a new log
I20260806 02:59:14.373953 4307 tablet_bootstrap.cc:492] T 1493ae7d57cc424092cb0bc35aa65ff0 P f1d23fb9d05d49debe91b1df45f52169: No bootstrap required, opened a new log
I20260806 02:59:14.374094 4309 tablet_bootstrap.cc:492] T 1493ae7d57cc424092cb0bc35aa65ff0 P a7137cef4cab448bb0d50e90c1342c5f: No bootstrap required, opened a new log
I20260806 02:59:14.374414 4308 ts_tablet_manager.cc:1397] T 1493ae7d57cc424092cb0bc35aa65ff0 P 6647d228143a4c3987f8b33924a6f317: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.002s
I20260806 02:59:14.374414 4307 ts_tablet_manager.cc:1397] T 1493ae7d57cc424092cb0bc35aa65ff0 P f1d23fb9d05d49debe91b1df45f52169: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.000s
I20260806 02:59:14.374578 4309 ts_tablet_manager.cc:1397] T 1493ae7d57cc424092cb0bc35aa65ff0 P a7137cef4cab448bb0d50e90c1342c5f: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.005s
I20260806 02:59:14.384953 4309 raft_consensus.cc:348] T 1493ae7d57cc424092cb0bc35aa65ff0 P a7137cef4cab448bb0d50e90c1342c5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6647d228143a4c3987f8b33924a6f317" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 45757 } } peers { permanent_uuid: "a7137cef4cab448bb0d50e90c1342c5f" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 40727 } } peers { permanent_uuid: "f1d23fb9d05d49debe91b1df45f52169" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 38641 } }
I20260806 02:59:14.385450 4309 raft_consensus.cc:374] T 1493ae7d57cc424092cb0bc35aa65ff0 P a7137cef4cab448bb0d50e90c1342c5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:59:14.385582 4309 raft_consensus.cc:729] T 1493ae7d57cc424092cb0bc35aa65ff0 P a7137cef4cab448bb0d50e90c1342c5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7137cef4cab448bb0d50e90c1342c5f, State: Initialized, Role: FOLLOWER
I20260806 02:59:14.386332 4309 consensus_queue.cc:260] T 1493ae7d57cc424092cb0bc35aa65ff0 P a7137cef4cab448bb0d50e90c1342c5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6647d228143a4c3987f8b33924a6f317" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 45757 } } peers { permanent_uuid: "a7137cef4cab448bb0d50e90c1342c5f" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 40727 } } peers { permanent_uuid: "f1d23fb9d05d49debe91b1df45f52169" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 38641 } }
I20260806 02:59:14.387719 4151 heartbeater.cc:503] Master 127.31.74.254:42793 was elected leader, sending a full tablet report...
I20260806 02:59:14.388103 4307 raft_consensus.cc:348] T 1493ae7d57cc424092cb0bc35aa65ff0 P f1d23fb9d05d49debe91b1df45f52169 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6647d228143a4c3987f8b33924a6f317" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 45757 } } peers { permanent_uuid: "a7137cef4cab448bb0d50e90c1342c5f" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 40727 } } peers { permanent_uuid: "f1d23fb9d05d49debe91b1df45f52169" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 38641 } }
I20260806 02:59:14.388088 4308 raft_consensus.cc:348] T 1493ae7d57cc424092cb0bc35aa65ff0 P 6647d228143a4c3987f8b33924a6f317 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6647d228143a4c3987f8b33924a6f317" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 45757 } } peers { permanent_uuid: "a7137cef4cab448bb0d50e90c1342c5f" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 40727 } } peers { permanent_uuid: "f1d23fb9d05d49debe91b1df45f52169" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 38641 } }
I20260806 02:59:14.388640 4307 raft_consensus.cc:374] T 1493ae7d57cc424092cb0bc35aa65ff0 P f1d23fb9d05d49debe91b1df45f52169 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:59:14.388653 4308 raft_consensus.cc:374] T 1493ae7d57cc424092cb0bc35aa65ff0 P 6647d228143a4c3987f8b33924a6f317 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:59:14.388761 4307 raft_consensus.cc:729] T 1493ae7d57cc424092cb0bc35aa65ff0 P f1d23fb9d05d49debe91b1df45f52169 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1d23fb9d05d49debe91b1df45f52169, State: Initialized, Role: FOLLOWER
I20260806 02:59:14.388761 4308 raft_consensus.cc:729] T 1493ae7d57cc424092cb0bc35aa65ff0 P 6647d228143a4c3987f8b33924a6f317 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6647d228143a4c3987f8b33924a6f317, State: Initialized, Role: FOLLOWER
I20260806 02:59:14.389034 4309 ts_tablet_manager.cc:1428] T 1493ae7d57cc424092cb0bc35aa65ff0 P a7137cef4cab448bb0d50e90c1342c5f: Time spent starting tablet: real 0.014s user 0.011s sys 0.003s
I20260806 02:59:14.389648 4307 consensus_queue.cc:260] T 1493ae7d57cc424092cb0bc35aa65ff0 P f1d23fb9d05d49debe91b1df45f52169 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6647d228143a4c3987f8b33924a6f317" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 45757 } } peers { permanent_uuid: "a7137cef4cab448bb0d50e90c1342c5f" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 40727 } } peers { permanent_uuid: "f1d23fb9d05d49debe91b1df45f52169" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 38641 } }
I20260806 02:59:14.389662 4308 consensus_queue.cc:260] T 1493ae7d57cc424092cb0bc35aa65ff0 P 6647d228143a4c3987f8b33924a6f317 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6647d228143a4c3987f8b33924a6f317" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 45757 } } peers { permanent_uuid: "a7137cef4cab448bb0d50e90c1342c5f" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 40727 } } peers { permanent_uuid: "f1d23fb9d05d49debe91b1df45f52169" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 38641 } }
I20260806 02:59:14.392832 4307 ts_tablet_manager.cc:1428] T 1493ae7d57cc424092cb0bc35aa65ff0 P f1d23fb9d05d49debe91b1df45f52169: Time spent starting tablet: real 0.018s user 0.011s sys 0.008s
I20260806 02:59:14.393802 4314 raft_consensus.cc:482] T 1493ae7d57cc424092cb0bc35aa65ff0 P 6647d228143a4c3987f8b33924a6f317 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:59:14.393918 4286 heartbeater.cc:503] Master 127.31.74.254:42793 was elected leader, sending a full tablet report...
I20260806 02:59:14.394261 4314 raft_consensus.cc:504] T 1493ae7d57cc424092cb0bc35aa65ff0 P 6647d228143a4c3987f8b33924a6f317 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6647d228143a4c3987f8b33924a6f317" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 45757 } } peers { permanent_uuid: "a7137cef4cab448bb0d50e90c1342c5f" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 40727 } } peers { permanent_uuid: "f1d23fb9d05d49debe91b1df45f52169" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 38641 } }
I20260806 02:59:14.394748 4308 ts_tablet_manager.cc:1428] T 1493ae7d57cc424092cb0bc35aa65ff0 P 6647d228143a4c3987f8b33924a6f317: Time spent starting tablet: real 0.020s user 0.022s sys 0.001s
I20260806 02:59:14.400835 4314 leader_election.cc:288] T 1493ae7d57cc424092cb0bc35aa65ff0 P 6647d228143a4c3987f8b33924a6f317 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a7137cef4cab448bb0d50e90c1342c5f (127.31.74.194:40727), f1d23fb9d05d49debe91b1df45f52169 (127.31.74.193:38641)
I20260806 02:59:14.405643 4304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:14.397757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.195:59379 (local address 127.31.74.194:40727)
0806 02:59:14.397960 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:14.397966 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:14.397985 (+ 19us) server_negotiation.cc:407] Connection header received
0806 02:59:14.398068 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:14.398075 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:14.398149 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:14.398245 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:59:14.398832 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:14.400136 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:14.402985 (+ 2849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:14.404416 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:14.404567 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:14.404628 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:14.405308 (+ 680us) server_negotiation.cc:299] Negotiation successful
0806 02:59:14.405433 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 02:59:14.405750 4290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:14.397279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.194:40727 (local address 127.31.74.195:59379)
0806 02:59:14.397507 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:14.397532 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:14.397667 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:14.398386 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:14.398391 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:14.398419 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:59:14.398656 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:14.398669 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:14.400360 (+ 1691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:14.400368 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:14.402809 (+ 2441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:14.402822 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:14.404676 (+ 1854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:14.404687 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:14.405006 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:14.405071 (+ 65us) client_negotiation.cc:769] Sending connection context
0806 02:59:14.405216 (+ 145us) client_negotiation.cc:240] Negotiation successful
0806 02:59:14.405384 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":1}
I20260806 02:59:14.406873 4104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1493ae7d57cc424092cb0bc35aa65ff0" candidate_uuid: "6647d228143a4c3987f8b33924a6f317" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7137cef4cab448bb0d50e90c1342c5f" is_pre_election: true
I20260806 02:59:14.407474 4104 raft_consensus.cc:2454] T 1493ae7d57cc424092cb0bc35aa65ff0 P a7137cef4cab448bb0d50e90c1342c5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6647d228143a4c3987f8b33924a6f317 in term 0.
I20260806 02:59:14.408419 4175 leader_election.cc:302] T 1493ae7d57cc424092cb0bc35aa65ff0 P 6647d228143a4c3987f8b33924a6f317 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6647d228143a4c3987f8b33924a6f317, a7137cef4cab448bb0d50e90c1342c5f; no voters:
I20260806 02:59:14.408957 4314 raft_consensus.cc:2781] T 1493ae7d57cc424092cb0bc35aa65ff0 P 6647d228143a4c3987f8b33924a6f317 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:59:14.409162 4314 raft_consensus.cc:482] T 1493ae7d57cc424092cb0bc35aa65ff0 P 6647d228143a4c3987f8b33924a6f317 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:59:14.409288 4314 raft_consensus.cc:3037] T 1493ae7d57cc424092cb0bc35aa65ff0 P 6647d228143a4c3987f8b33924a6f317 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:59:14.409614 4306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:14.398026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.195:58527 (local address 127.31.74.193:38641)
0806 02:59:14.398230 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:14.398238 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:14.400978 (+ 2740us) server_negotiation.cc:407] Connection header received
0806 02:59:14.401302 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:14.401311 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:14.401419 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:14.401494 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:59:14.402432 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:14.403675 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:14.407148 (+ 3473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:14.408262 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:14.408473 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:14.408552 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:14.409139 (+ 587us) server_negotiation.cc:299] Negotiation successful
0806 02:59:14.409346 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 02:59:14.409948 4317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:14.398069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:38641 (local address 127.31.74.195:58527)
0806 02:59:14.400863 (+ 2794us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:14.400885 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:14.401098 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:14.401713 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:14.401718 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:14.401746 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:59:14.402243 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:14.402255 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:14.404144 (+ 1889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:14.404159 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:14.406679 (+ 2520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:14.406702 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:14.408546 (+ 1844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:14.408556 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:14.408784 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:14.408888 (+ 104us) client_negotiation.cc:769] Sending connection context
0806 02:59:14.409024 (+ 136us) client_negotiation.cc:240] Negotiation successful
0806 02:59:14.409223 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2635,"thread_start_us":2441,"threads_started":1}
I20260806 02:59:14.411484 3967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1493ae7d57cc424092cb0bc35aa65ff0" candidate_uuid: "6647d228143a4c3987f8b33924a6f317" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1d23fb9d05d49debe91b1df45f52169" is_pre_election: true
I20260806 02:59:14.412154 3967 raft_consensus.cc:2454] T 1493ae7d57cc424092cb0bc35aa65ff0 P f1d23fb9d05d49debe91b1df45f52169 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6647d228143a4c3987f8b33924a6f317 in term 0.
I20260806 02:59:14.413722 4314 raft_consensus.cc:504] T 1493ae7d57cc424092cb0bc35aa65ff0 P 6647d228143a4c3987f8b33924a6f317 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6647d228143a4c3987f8b33924a6f317" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 45757 } } peers { permanent_uuid: "a7137cef4cab448bb0d50e90c1342c5f" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 40727 } } peers { permanent_uuid: "f1d23fb9d05d49debe91b1df45f52169" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 38641 } }
I20260806 02:59:14.414467 4314 leader_election.cc:288] T 1493ae7d57cc424092cb0bc35aa65ff0 P 6647d228143a4c3987f8b33924a6f317 [CANDIDATE]: Term 1 election: Requested vote from peers a7137cef4cab448bb0d50e90c1342c5f (127.31.74.194:40727), f1d23fb9d05d49debe91b1df45f52169 (127.31.74.193:38641)
I20260806 02:59:14.415143 3967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1493ae7d57cc424092cb0bc35aa65ff0" candidate_uuid: "6647d228143a4c3987f8b33924a6f317" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1d23fb9d05d49debe91b1df45f52169"
I20260806 02:59:14.415234 4104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1493ae7d57cc424092cb0bc35aa65ff0" candidate_uuid: "6647d228143a4c3987f8b33924a6f317" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7137cef4cab448bb0d50e90c1342c5f"
I20260806 02:59:14.415416 3967 raft_consensus.cc:3037] T 1493ae7d57cc424092cb0bc35aa65ff0 P f1d23fb9d05d49debe91b1df45f52169 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:59:14.415560 4104 raft_consensus.cc:3037] T 1493ae7d57cc424092cb0bc35aa65ff0 P a7137cef4cab448bb0d50e90c1342c5f [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:59:14.418143 3967 raft_consensus.cc:2454] T 1493ae7d57cc424092cb0bc35aa65ff0 P f1d23fb9d05d49debe91b1df45f52169 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6647d228143a4c3987f8b33924a6f317 in term 1.
I20260806 02:59:14.418736 4173 leader_election.cc:302] T 1493ae7d57cc424092cb0bc35aa65ff0 P 6647d228143a4c3987f8b33924a6f317 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6647d228143a4c3987f8b33924a6f317, f1d23fb9d05d49debe91b1df45f52169; no voters:
I20260806 02:59:14.419374 4314 raft_consensus.cc:2781] T 1493ae7d57cc424092cb0bc35aa65ff0 P 6647d228143a4c3987f8b33924a6f317 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:59:14.419884 4104 raft_consensus.cc:2454] T 1493ae7d57cc424092cb0bc35aa65ff0 P a7137cef4cab448bb0d50e90c1342c5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6647d228143a4c3987f8b33924a6f317 in term 1.
I20260806 02:59:14.419763 4314 raft_consensus.cc:686] T 1493ae7d57cc424092cb0bc35aa65ff0 P 6647d228143a4c3987f8b33924a6f317 [term 1 LEADER]: Becoming Leader. State: Replica: 6647d228143a4c3987f8b33924a6f317, State: Running, Role: LEADER
I20260806 02:59:14.421643 4314 consensus_queue.cc:237] T 1493ae7d57cc424092cb0bc35aa65ff0 P 6647d228143a4c3987f8b33924a6f317 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6647d228143a4c3987f8b33924a6f317" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 45757 } } peers { permanent_uuid: "a7137cef4cab448bb0d50e90c1342c5f" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 40727 } } peers { permanent_uuid: "f1d23fb9d05d49debe91b1df45f52169" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 38641 } }
I20260806 02:59:14.428220 3821 catalog_manager.cc:5495] T 1493ae7d57cc424092cb0bc35aa65ff0 P 6647d228143a4c3987f8b33924a6f317 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6647d228143a4c3987f8b33924a6f317 (127.31.74.195). New cstate: current_term: 1 leader_uuid: "6647d228143a4c3987f8b33924a6f317" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6647d228143a4c3987f8b33924a6f317" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 45757 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a7137cef4cab448bb0d50e90c1342c5f" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 40727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1d23fb9d05d49debe91b1df45f52169" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 38641 } health_report { overall_health: UNKNOWN } } }
I20260806 02:59:14.458173 4303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:14.450880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60538 (local address 127.31.74.195:45757)
0806 02:59:14.451178 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:14.451183 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:14.451204 (+ 21us) server_negotiation.cc:407] Connection header received
0806 02:59:14.451490 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:14.451495 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:14.451609 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:14.451782 (+ 173us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:59:14.452401 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:14.453138 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:14.454660 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:14.455152 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:14.455273 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:14.456207 (+ 934us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:59:14.457074 (+ 867us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:59:14.457386 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:14.457656 (+ 270us) server_negotiation.cc:299] Negotiation successful
0806 02:59:14.457770 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260806 02:59:14.470110 3967 raft_consensus.cc:1261] T 1493ae7d57cc424092cb0bc35aa65ff0 P f1d23fb9d05d49debe91b1df45f52169 [term 1 FOLLOWER]: Refusing update from remote peer 6647d228143a4c3987f8b33924a6f317: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:59:14.470649 4104 raft_consensus.cc:1261] T 1493ae7d57cc424092cb0bc35aa65ff0 P a7137cef4cab448bb0d50e90c1342c5f [term 1 FOLLOWER]: Refusing update from remote peer 6647d228143a4c3987f8b33924a6f317: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260806 02:59:14.470709 4287 tablet.cc:2369] T 1493ae7d57cc424092cb0bc35aa65ff0 P 6647d228143a4c3987f8b33924a6f317: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:59:14.471874 4316 consensus_queue.cc:1035] T 1493ae7d57cc424092cb0bc35aa65ff0 P 6647d228143a4c3987f8b33924a6f317 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1d23fb9d05d49debe91b1df45f52169" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 38641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:59:14.472314 4314 consensus_queue.cc:1035] T 1493ae7d57cc424092cb0bc35aa65ff0 P 6647d228143a4c3987f8b33924a6f317 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7137cef4cab448bb0d50e90c1342c5f" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 40727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:59:14.508953 3821 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58606:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:59:14.509297 3821 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58606:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:59:14.515786 3821 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dd161421a51e44f2a55eb59929352a07: Sending DeleteTablet for 3 replicas of tablet 1493ae7d57cc424092cb0bc35aa65ff0
I20260806 02:59:14.517318 4084 tablet_service.cc:1514] Processing DeleteTablet for tablet 1493ae7d57cc424092cb0bc35aa65ff0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:59:14 UTC) from {username='kudu', principal='kudu/127.31.74.254@KRBTEST.COM'} at 127.0.0.1:59738
I20260806 02:59:14.517369 4221 tablet_service.cc:1514] Processing DeleteTablet for tablet 1493ae7d57cc424092cb0bc35aa65ff0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:59:14 UTC) from {username='kudu', principal='kudu/127.31.74.254@KRBTEST.COM'} at 127.0.0.1:60522
I20260806 02:59:14.518496 3947 tablet_service.cc:1514] Processing DeleteTablet for tablet 1493ae7d57cc424092cb0bc35aa65ff0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:59:14 UTC) from {username='kudu', principal='kudu/127.31.74.254@KRBTEST.COM'} at 127.0.0.1:48996
I20260806 02:59:14.520980 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 3883
I20260806 02:59:14.533839 4332 tablet_replica.cc:330] T 1493ae7d57cc424092cb0bc35aa65ff0 P f1d23fb9d05d49debe91b1df45f52169: stopping tablet replica
I20260806 02:59:14.535665 4332 raft_consensus.cc:2229] T 1493ae7d57cc424092cb0bc35aa65ff0 P f1d23fb9d05d49debe91b1df45f52169 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:59:14.537518 4332 raft_consensus.cc:2258] T 1493ae7d57cc424092cb0bc35aa65ff0 P f1d23fb9d05d49debe91b1df45f52169 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:59:14.539958 4331 tablet_replica.cc:330] T 1493ae7d57cc424092cb0bc35aa65ff0 P 6647d228143a4c3987f8b33924a6f317: stopping tablet replica
I20260806 02:59:14.540665 4331 raft_consensus.cc:2229] T 1493ae7d57cc424092cb0bc35aa65ff0 P 6647d228143a4c3987f8b33924a6f317 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:59:14.542172 4331 raft_consensus.cc:2258] T 1493ae7d57cc424092cb0bc35aa65ff0 P 6647d228143a4c3987f8b33924a6f317 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:59:14.542281 4332 ts_tablet_manager.cc:1905] T 1493ae7d57cc424092cb0bc35aa65ff0 P f1d23fb9d05d49debe91b1df45f52169: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 06 02:59:14 dist-test-slave-dhph krb5kdc[3777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985151, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.74.193@KRBTEST.COM
I20260806 02:59:14.545264 4331 ts_tablet_manager.cc:1905] T 1493ae7d57cc424092cb0bc35aa65ff0 P 6647d228143a4c3987f8b33924a6f317: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:59:14.545684 4330 tablet_replica.cc:330] T 1493ae7d57cc424092cb0bc35aa65ff0 P a7137cef4cab448bb0d50e90c1342c5f: stopping tablet replica
I20260806 02:59:14.554752 4332 ts_tablet_manager.cc:1918] T 1493ae7d57cc424092cb0bc35aa65ff0 P f1d23fb9d05d49debe91b1df45f52169: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:59:14.554757 4331 ts_tablet_manager.cc:1918] T 1493ae7d57cc424092cb0bc35aa65ff0 P 6647d228143a4c3987f8b33924a6f317: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:59:14.554996 4331 log.cc:1196] T 1493ae7d57cc424092cb0bc35aa65ff0 P 6647d228143a4c3987f8b33924a6f317: Deleting WAL directory at /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/minicluster-data/ts-2/wal/wals/1493ae7d57cc424092cb0bc35aa65ff0
I20260806 02:59:14.555181 4332 log.cc:1196] T 1493ae7d57cc424092cb0bc35aa65ff0 P f1d23fb9d05d49debe91b1df45f52169: Deleting WAL directory at /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/minicluster-data/ts-0/wal/wals/1493ae7d57cc424092cb0bc35aa65ff0
I20260806 02:59:14.556119 4332 ts_tablet_manager.cc:1939] T 1493ae7d57cc424092cb0bc35aa65ff0 P f1d23fb9d05d49debe91b1df45f52169: Deleting consensus metadata
I20260806 02:59:14.556012 4331 ts_tablet_manager.cc:1939] T 1493ae7d57cc424092cb0bc35aa65ff0 P 6647d228143a4c3987f8b33924a6f317: Deleting consensus metadata
I20260806 02:59:14.558717 3807 catalog_manager.cc:4841] TS f1d23fb9d05d49debe91b1df45f52169 (127.31.74.193:38641): tablet 1493ae7d57cc424092cb0bc35aa65ff0 (table test-table [id=1978fe62e881430aab4d8e993e60ae43]) successfully deleted
I20260806 02:59:14.559130 3807 catalog_manager.cc:4841] TS 6647d228143a4c3987f8b33924a6f317 (127.31.74.195:45757): tablet 1493ae7d57cc424092cb0bc35aa65ff0 (table test-table [id=1978fe62e881430aab4d8e993e60ae43]) successfully deleted
I20260806 02:59:14.559357 4330 raft_consensus.cc:2229] T 1493ae7d57cc424092cb0bc35aa65ff0 P a7137cef4cab448bb0d50e90c1342c5f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:59:14.560103 4330 raft_consensus.cc:2258] T 1493ae7d57cc424092cb0bc35aa65ff0 P a7137cef4cab448bb0d50e90c1342c5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:59:14.563408 4330 ts_tablet_manager.cc:1905] T 1493ae7d57cc424092cb0bc35aa65ff0 P a7137cef4cab448bb0d50e90c1342c5f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:59:14.564875 4306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:14.524028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48998 (local address 127.31.74.193:38641)
0806 02:59:14.524309 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:14.524316 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:14.527554 (+ 3238us) server_negotiation.cc:407] Connection header received
0806 02:59:14.529429 (+ 1875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:14.529439 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:14.529572 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:14.529831 (+ 259us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:14.533534 (+ 3703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:14.534857 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:14.536966 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:14.537735 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:14.537851 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:14.545408 (+ 7557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:14.545427 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:14.545447 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:14.545506 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:14.550488 (+ 4982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:14.560935 (+ 10447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:14.560949 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:14.560952 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:14.561437 (+ 485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:14.562144 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:14.562151 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:14.562154 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:14.562812 (+ 658us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:14.563204 (+ 392us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:14.563984 (+ 780us) server_negotiation.cc:299] Negotiation successful
0806 02:59:14.564359 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260806 02:59:14.566992 4008 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:59:14.577200 4330 ts_tablet_manager.cc:1918] T 1493ae7d57cc424092cb0bc35aa65ff0 P a7137cef4cab448bb0d50e90c1342c5f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:59:14.577608 4330 log.cc:1196] T 1493ae7d57cc424092cb0bc35aa65ff0 P a7137cef4cab448bb0d50e90c1342c5f: Deleting WAL directory at /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785985105255475-32043-0/minicluster-data/ts-1/wal/wals/1493ae7d57cc424092cb0bc35aa65ff0
I20260806 02:59:14.578549 4330 ts_tablet_manager.cc:1939] T 1493ae7d57cc424092cb0bc35aa65ff0 P a7137cef4cab448bb0d50e90c1342c5f: Deleting consensus metadata
I20260806 02:59:14.581676 3809 catalog_manager.cc:4841] TS a7137cef4cab448bb0d50e90c1342c5f (127.31.74.194:40727): tablet 1493ae7d57cc424092cb0bc35aa65ff0 (table test-table [id=1978fe62e881430aab4d8e993e60ae43]) successfully deleted
I20260806 02:59:14.708351 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 3883
I20260806 02:59:14.734869 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 4019
Aug 06 02:59:14 dist-test-slave-dhph krb5kdc[3777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985151, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.74.194@KRBTEST.COM
I20260806 02:59:14.758498 4304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:14.735998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59740 (local address 127.31.74.194:40727)
0806 02:59:14.736232 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:14.736238 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:14.736261 (+ 23us) server_negotiation.cc:407] Connection header received
0806 02:59:14.736386 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:14.736390 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:14.736456 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:14.736567 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:14.737932 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:14.738861 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:14.740992 (+ 2131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:14.741627 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:14.741881 (+ 254us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:14.747218 (+ 5337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:14.747246 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:14.747269 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:14.747339 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:14.752543 (+ 5204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:14.754238 (+ 1695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:14.754255 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:14.754259 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:14.754677 (+ 418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:14.755418 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:14.755427 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:14.755431 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:14.756247 (+ 816us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:14.757067 (+ 820us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:14.757597 (+ 530us) server_negotiation.cc:299] Negotiation successful
0806 02:59:14.757930 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260806 02:59:14.759564 4146 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:59:14.870324 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 4019
I20260806 02:59:14.900946 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 4157
Aug 06 02:59:14 dist-test-slave-dhph krb5kdc[3777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985151, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.74.195@KRBTEST.COM
I20260806 02:59:14.924186 4303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:14.901970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60542 (local address 127.31.74.195:45757)
0806 02:59:14.902209 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:14.902216 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:14.902240 (+ 24us) server_negotiation.cc:407] Connection header received
0806 02:59:14.902396 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:14.902403 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:14.902503 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:14.902686 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:14.904446 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:14.905856 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:14.907568 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:14.908330 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:14.908502 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:14.913959 (+ 5457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:14.913980 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:14.914002 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:14.914077 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:14.920214 (+ 6137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:14.921308 (+ 1094us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:14.921322 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:14.921328 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:14.921560 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:14.922091 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:14.922102 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:14.922108 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:14.922739 (+ 631us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:14.923115 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:14.923415 (+ 300us) server_negotiation.cc:299] Negotiation successful
0806 02:59:14.923706 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260806 02:59:14.925561 4281 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:59:15.066094 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 4157
I20260806 02:59:15.092506 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 3790
I20260806 02:59:15.094926 3852 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:59:15.222956 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 3790
2026-08-06T02:59:15Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (5606 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-06T02:59:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:59:15Z Disabled control of system clock
I20260806 02:59:15.311548 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785985105255475-32043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785985105255475-32043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785985105255475-32043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:41165
--webserver_interface=127.31.74.254
--webserver_port=0
--builtin_ntp_servers=127.31.74.212:33911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.74.254:41165
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260806 02:59:15.702111 4341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:59:15.702454 4341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:59:15.702711 4341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:59:15.712064 4341 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:59:15.712210 4341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:59:15.712311 4341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:59:15.712428 4341 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:59:15.712518 4341 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:59:15.723440 4341 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:33911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785985105255475-32043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785985105255475-32043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.74.254:41165
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:41165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.74.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785985105255475-32043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:59:15.725150 4341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:59:15.728089 4341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:59:15.742642 4349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:15.742635 4346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:15.742897 4347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:59:15.744261 4341 server_base.cc:1034] running on GCE node
I20260806 02:59:15.745213 4341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:59:15.746883 4341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:59:15.748375 4341 hybrid_clock.cc:648] HybridClock initialized: now 1785985155748133 us; error 50 us; skew 500 ppm
I20260806 02:59:15.748843 4341 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 02:59:15.751937 4341 webserver.cc:460] Webserver started at http://127.31.74.254:33747/ using document root <none> and password file <none>
I20260806 02:59:15.752777 4341 fs_manager.cc:359] Metadata directory not provided
I20260806 02:59:15.752938 4341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:59:15.753342 4341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:59:15.756682 4341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785985105255475-32043-0/minicluster-data/master-0/data/instance:
uuid: "54cb2d1b3cb0430fbe9a6fccc870fbb3"
format_stamp: "Formatted at 2026-08-06 02:59:15 on dist-test-slave-dhph"
server_key: "7b8f63d6df70a7356a5a50d3d3747b27"
server_key_iv: "11b3bf1729ecdf84d1ea6507e5a9d6b8"
server_key_version: "encryptionkey@0"
I20260806 02:59:15.757778 4341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785985105255475-32043-0/minicluster-data/master-0/wal/instance:
uuid: "54cb2d1b3cb0430fbe9a6fccc870fbb3"
format_stamp: "Formatted at 2026-08-06 02:59:15 on dist-test-slave-dhph"
server_key: "7b8f63d6df70a7356a5a50d3d3747b27"
server_key_iv: "11b3bf1729ecdf84d1ea6507e5a9d6b8"
server_key_version: "encryptionkey@0"
I20260806 02:59:15.764950 4341 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20260806 02:59:15.771008 4355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:15.773054 4341 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260806 02:59:15.773319 4341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785985105255475-32043-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785985105255475-32043-0/minicluster-data/master-0/wal
uuid: "54cb2d1b3cb0430fbe9a6fccc870fbb3"
format_stamp: "Formatted at 2026-08-06 02:59:15 on dist-test-slave-dhph"
server_key: "7b8f63d6df70a7356a5a50d3d3747b27"
server_key_iv: "11b3bf1729ecdf84d1ea6507e5a9d6b8"
server_key_version: "encryptionkey@0"
I20260806 02:59:15.773607 4341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785985105255475-32043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785985105255475-32043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785985105255475-32043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:59:15.781189 4341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:59:15.782382 4341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:59:15.782840 4341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:59:15.808012 4341 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.254:41165
I20260806 02:59:15.808040 4406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.254:41165 every 8 connection(s)
I20260806 02:59:15.810176 4341 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
I20260806 02:59:15.812096 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 4341
I20260806 02:59:15.812314 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785985105255475-32043-0/minicluster-data/master-0/wal/instance
I20260806 02:59:15.813068 32043 external_mini_cluster.cc:1428] Setting key 51a549fcf55a8d1f40707af9f95e510d
I20260806 02:59:15.814662 4407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:59:15.822968 4410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:15.815539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43888 (local address 127.31.74.254:41165)
0806 02:59:15.816725 (+ 1186us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:15.816736 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:15.816784 (+ 48us) server_negotiation.cc:407] Connection header received
0806 02:59:15.818481 (+ 1697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:15.818504 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:15.819172 (+ 668us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:15.819627 (+ 455us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:15.820899 (+ 1272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:15.820906 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:15.820919 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:59:15.820936 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:15.821020 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:59:15.821037 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:15.821321 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:15.821514 (+ 193us) server_negotiation.cc:299] Negotiation successful
0806 02:59:15.821858 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":734,"thread_start_us":500,"threads_started":1}
I20260806 02:59:15.830041 4407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54cb2d1b3cb0430fbe9a6fccc870fbb3: Bootstrap starting.
I20260806 02:59:15.835618 4407 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54cb2d1b3cb0430fbe9a6fccc870fbb3: Neither blocks nor log segments found. Creating new log.
I20260806 02:59:15.837152 4407 log.cc:824] T 00000000000000000000000000000000 P 54cb2d1b3cb0430fbe9a6fccc870fbb3: Log is configured to *not* fsync() on all Append() calls
I20260806 02:59:15.841298 4407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54cb2d1b3cb0430fbe9a6fccc870fbb3: No bootstrap required, opened a new log
I20260806 02:59:15.852490 4407 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54cb2d1b3cb0430fbe9a6fccc870fbb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54cb2d1b3cb0430fbe9a6fccc870fbb3" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 41165 } }
I20260806 02:59:15.853130 4407 raft_consensus.cc:374] T 00000000000000000000000000000000 P 54cb2d1b3cb0430fbe9a6fccc870fbb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:59:15.853289 4407 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54cb2d1b3cb0430fbe9a6fccc870fbb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54cb2d1b3cb0430fbe9a6fccc870fbb3, State: Initialized, Role: FOLLOWER
I20260806 02:59:15.854349 4407 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54cb2d1b3cb0430fbe9a6fccc870fbb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54cb2d1b3cb0430fbe9a6fccc870fbb3" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 41165 } }
I20260806 02:59:15.854761 4407 raft_consensus.cc:388] T 00000000000000000000000000000000 P 54cb2d1b3cb0430fbe9a6fccc870fbb3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:59:15.854988 4407 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54cb2d1b3cb0430fbe9a6fccc870fbb3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:59:15.855202 4407 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54cb2d1b3cb0430fbe9a6fccc870fbb3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:59:15.858932 4407 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54cb2d1b3cb0430fbe9a6fccc870fbb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54cb2d1b3cb0430fbe9a6fccc870fbb3" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 41165 } }
I20260806 02:59:15.859795 4407 leader_election.cc:302] T 00000000000000000000000000000000 P 54cb2d1b3cb0430fbe9a6fccc870fbb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 54cb2d1b3cb0430fbe9a6fccc870fbb3; no voters:
I20260806 02:59:15.861594 4407 leader_election.cc:288] T 00000000000000000000000000000000 P 54cb2d1b3cb0430fbe9a6fccc870fbb3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:59:15.861802 4412 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 54cb2d1b3cb0430fbe9a6fccc870fbb3 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:59:15.863293 4412 raft_consensus.cc:686] T 00000000000000000000000000000000 P 54cb2d1b3cb0430fbe9a6fccc870fbb3 [term 1 LEADER]: Becoming Leader. State: Replica: 54cb2d1b3cb0430fbe9a6fccc870fbb3, State: Running, Role: LEADER
I20260806 02:59:15.864655 4412 consensus_queue.cc:237] T 00000000000000000000000000000000 P 54cb2d1b3cb0430fbe9a6fccc870fbb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54cb2d1b3cb0430fbe9a6fccc870fbb3" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 41165 } }
I20260806 02:59:15.866101 4407 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54cb2d1b3cb0430fbe9a6fccc870fbb3 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:59:15.869858 4413 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54cb2d1b3cb0430fbe9a6fccc870fbb3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "54cb2d1b3cb0430fbe9a6fccc870fbb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54cb2d1b3cb0430fbe9a6fccc870fbb3" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 41165 } } }
I20260806 02:59:15.872400 4413 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54cb2d1b3cb0430fbe9a6fccc870fbb3 [sys.catalog]: This master's current role is: LEADER
I20260806 02:59:15.874358 4419 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:59:15.871995 4414 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54cb2d1b3cb0430fbe9a6fccc870fbb3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54cb2d1b3cb0430fbe9a6fccc870fbb3. Latest consensus state: current_term: 1 leader_uuid: "54cb2d1b3cb0430fbe9a6fccc870fbb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54cb2d1b3cb0430fbe9a6fccc870fbb3" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 41165 } } }
I20260806 02:59:15.875252 4414 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54cb2d1b3cb0430fbe9a6fccc870fbb3 [sys.catalog]: This master's current role is: LEADER
I20260806 02:59:15.884063 4419 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:59:15.896632 4419 catalog_manager.cc:1347] Generated new cluster ID: 12e7f59f99a34da28147bc00a25088ac
I20260806 02:59:15.896797 4419 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:59:15.921680 4419 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:59:15.922315 4419 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:59:15.935750 4419 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 54cb2d1b3cb0430fbe9a6fccc870fbb3: Generated new TSK 0
I20260806 02:59:15.936904 4419 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:59:15.951785 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785985105255475-32043-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785985105255475-32043-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785985105255475-32043-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.193:0
--local_ip_for_outbound_sockets=127.31.74.193
--webserver_interface=127.31.74.193
--webserver_port=0
--tserver_master_addrs=127.31.74.254:41165
--builtin_ntp_servers=127.31.74.212:33911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260806 02:59:16.408703 4432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:59:16.409461 4432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:59:16.409654 4432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:59:16.418982 4432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:59:16.419173 4432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:59:16.419394 4432 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.193
I20260806 02:59:16.431173 4432 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:33911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785985105255475-32043-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785985105255475-32043-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.74.193
--webserver_port=0
--tserver_master_addrs=127.31.74.254:41165
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.193
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785985105255475-32043-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:59:16.432967 4432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:59:16.435364 4432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:59:16.453099 4438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:16.453193 4440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:16.453058 4437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:59:16.454047 4432 server_base.cc:1034] running on GCE node
I20260806 02:59:16.454949 4432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:59:16.456178 4432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:59:16.457475 4432 hybrid_clock.cc:648] HybridClock initialized: now 1785985156457409 us; error 55 us; skew 500 ppm
I20260806 02:59:16.457965 4432 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 02:59:16.461704 4432 webserver.cc:460] Webserver started at http://127.31.74.193:41275/ using document root <none> and password file <none>
I20260806 02:59:16.462734 4432 fs_manager.cc:359] Metadata directory not provided
I20260806 02:59:16.462916 4432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:59:16.463264 4432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:59:16.467547 4432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785985105255475-32043-0/minicluster-data/ts-0/data/instance:
uuid: "ccc5e8406a4d45349fbc6ca8d4a65108"
format_stamp: "Formatted at 2026-08-06 02:59:16 on dist-test-slave-dhph"
server_key: "5512ff08c9f883198c28b6fe126fc248"
server_key_iv: "0bb5d76c63dcac6cc745c5633bf3b80b"
server_key_version: "encryptionkey@0"
I20260806 02:59:16.469058 4432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785985105255475-32043-0/minicluster-data/ts-0/wal/instance:
uuid: "ccc5e8406a4d45349fbc6ca8d4a65108"
format_stamp: "Formatted at 2026-08-06 02:59:16 on dist-test-slave-dhph"
server_key: "5512ff08c9f883198c28b6fe126fc248"
server_key_iv: "0bb5d76c63dcac6cc745c5633bf3b80b"
server_key_version: "encryptionkey@0"
I20260806 02:59:16.476794 4432 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20260806 02:59:16.481822 4446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:16.483892 4432 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260806 02:59:16.484103 4432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785985105255475-32043-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785985105255475-32043-0/minicluster-data/ts-0/wal
uuid: "ccc5e8406a4d45349fbc6ca8d4a65108"
format_stamp: "Formatted at 2026-08-06 02:59:16 on dist-test-slave-dhph"
server_key: "5512ff08c9f883198c28b6fe126fc248"
server_key_iv: "0bb5d76c63dcac6cc745c5633bf3b80b"
server_key_version: "encryptionkey@0"
I20260806 02:59:16.484382 4432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785985105255475-32043-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785985105255475-32043-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785985105255475-32043-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:59:16.493368 4432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:59:16.494372 4432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:59:16.494810 4432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:59:16.495971 4432 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:59:16.497998 4432 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:59:16.498162 4432 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:16.498315 4432 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:59:16.498418 4432 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:16.544006 4432 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.193:41077
I20260806 02:59:16.544065 4559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.193:41077 every 8 connection(s)
I20260806 02:59:16.545845 4432 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
I20260806 02:59:16.550853 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 4432
I20260806 02:59:16.551091 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785985105255475-32043-0/minicluster-data/ts-0/wal/instance
I20260806 02:59:16.552054 32043 external_mini_cluster.cc:1428] Setting key 7f38d522e3d2a933a6029cd43845e862
I20260806 02:59:16.556867 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785985105255475-32043-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785985105255475-32043-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785985105255475-32043-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.194:0
--local_ip_for_outbound_sockets=127.31.74.194
--webserver_interface=127.31.74.194
--webserver_port=0
--tserver_master_addrs=127.31.74.254:41165
--builtin_ntp_servers=127.31.74.212:33911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260806 02:59:16.557716 4562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:16.548954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.193:43139 (local address 127.31.74.254:41165)
0806 02:59:16.549478 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:16.549483 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:16.551072 (+ 1589us) server_negotiation.cc:407] Connection header received
0806 02:59:16.554103 (+ 3031us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:16.554110 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:16.554206 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:16.554652 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:16.556832 (+ 2180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:16.556837 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:16.556841 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:59:16.556860 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:16.556911 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:59:16.556925 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:16.557101 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:16.557333 (+ 232us) server_negotiation.cc:299] Negotiation successful
0806 02:59:16.557413 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":296,"threads_started":1}
I20260806 02:59:16.558975 4563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:16.549435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:41165 (local address 127.31.74.193:43139)
0806 02:59:16.550839 (+ 1404us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:16.550898 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:16.553659 (+ 2761us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:16.555207 (+ 1548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:16.555222 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:16.555383 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:59:16.556332 (+ 949us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:59:16.556336 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:59:16.556402 (+ 66us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:59:16.556548 (+ 146us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:59:16.557067 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:59:16.557080 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:59:16.557089 (+ 9us) client_negotiation.cc:769] Sending connection context
0806 02:59:16.557479 (+ 390us) client_negotiation.cc:240] Negotiation successful
0806 02:59:16.557806 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":700,"thread_start_us":338,"threads_started":1}
I20260806 02:59:16.560896 4560 heartbeater.cc:347] Connected to a master server at 127.31.74.254:41165
I20260806 02:59:16.561234 4560 heartbeater.cc:464] Registering TS with master...
I20260806 02:59:16.561916 4560 heartbeater.cc:511] Master 127.31.74.254:41165 requested a full tablet report, sending...
I20260806 02:59:16.564718 4372 ts_manager.cc:194] Registered new tserver with Master: ccc5e8406a4d45349fbc6ca8d4a65108 (127.31.74.193:41077)
W20260806 02:59:17.002425 4564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:59:17.002923 4564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:59:17.003108 4564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:59:17.012557 4564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:59:17.012741 4564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:59:17.012991 4564 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.194
I20260806 02:59:17.025733 4564 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:33911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785985105255475-32043-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785985105255475-32043-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.74.194
--webserver_port=0
--tserver_master_addrs=127.31.74.254:41165
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.194
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785985105255475-32043-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:59:17.027346 4564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:59:17.029277 4564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:59:17.045493 4569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:17.046496 4571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:17.047027 4573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:59:17.048769 4564 server_base.cc:1034] running on GCE node
I20260806 02:59:17.049484 4564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:59:17.050889 4564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:59:17.052160 4564 hybrid_clock.cc:648] HybridClock initialized: now 1785985157052089 us; error 64 us; skew 500 ppm
I20260806 02:59:17.052624 4564 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 02:59:17.062744 4564 webserver.cc:460] Webserver started at http://127.31.74.194:33229/ using document root <none> and password file <none>
I20260806 02:59:17.063928 4564 fs_manager.cc:359] Metadata directory not provided
I20260806 02:59:17.064064 4564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:59:17.064735 4564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:59:17.069188 4564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785985105255475-32043-0/minicluster-data/ts-1/data/instance:
uuid: "82093717a7424668afbb0d6aac0c7596"
format_stamp: "Formatted at 2026-08-06 02:59:17 on dist-test-slave-dhph"
server_key: "fd9df05e11b40507ec78e190375e40ff"
server_key_iv: "4ddbe9d30c509fa1a80c57fef7bf8675"
server_key_version: "encryptionkey@0"
I20260806 02:59:17.070295 4564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785985105255475-32043-0/minicluster-data/ts-1/wal/instance:
uuid: "82093717a7424668afbb0d6aac0c7596"
format_stamp: "Formatted at 2026-08-06 02:59:17 on dist-test-slave-dhph"
server_key: "fd9df05e11b40507ec78e190375e40ff"
server_key_iv: "4ddbe9d30c509fa1a80c57fef7bf8675"
server_key_version: "encryptionkey@0"
I20260806 02:59:17.080209 4564 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.007s sys 0.000s
I20260806 02:59:17.085390 4579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:17.087630 4564 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260806 02:59:17.087886 4564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785985105255475-32043-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785985105255475-32043-0/minicluster-data/ts-1/wal
uuid: "82093717a7424668afbb0d6aac0c7596"
format_stamp: "Formatted at 2026-08-06 02:59:17 on dist-test-slave-dhph"
server_key: "fd9df05e11b40507ec78e190375e40ff"
server_key_iv: "4ddbe9d30c509fa1a80c57fef7bf8675"
server_key_version: "encryptionkey@0"
I20260806 02:59:17.088171 4564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785985105255475-32043-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785985105255475-32043-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785985105255475-32043-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:59:17.095759 4564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:59:17.096776 4564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:59:17.097110 4564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:59:17.098344 4564 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:59:17.100235 4564 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:59:17.100375 4564 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:17.100548 4564 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:59:17.100653 4564 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:17.144943 4564 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.194:37741
I20260806 02:59:17.144999 4692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.194:37741 every 8 connection(s)
I20260806 02:59:17.146474 4564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
I20260806 02:59:17.155643 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 4564
I20260806 02:59:17.155856 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785985105255475-32043-0/minicluster-data/ts-1/wal/instance
I20260806 02:59:17.156530 32043 external_mini_cluster.cc:1428] Setting key d7b7da743b9e2f2dc652cbba1d746ad5
I20260806 02:59:17.157761 4695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:17.150090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.194:47733 (local address 127.31.74.254:41165)
0806 02:59:17.150764 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:17.150772 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:17.152398 (+ 1626us) server_negotiation.cc:407] Connection header received
0806 02:59:17.154496 (+ 2098us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:17.154503 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:17.154569 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:17.154790 (+ 221us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:17.156793 (+ 2003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:17.156798 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:17.156802 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:59:17.156821 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:17.156870 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:59:17.156883 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:17.157008 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:17.157361 (+ 353us) server_negotiation.cc:299] Negotiation successful
0806 02:59:17.157449 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":317,"threads_started":1}
I20260806 02:59:17.159238 4696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:17.150704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:41165 (local address 127.31.74.194:47733)
0806 02:59:17.152134 (+ 1430us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:17.152195 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:17.154130 (+ 1935us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:17.155187 (+ 1057us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:17.155200 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:17.155359 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:59:17.156326 (+ 967us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:59:17.156331 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:59:17.156391 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:59:17.156526 (+ 135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:59:17.157007 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:59:17.157019 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:59:17.157027 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 02:59:17.157415 (+ 388us) client_negotiation.cc:240] Negotiation successful
0806 02:59:17.157807 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":781,"thread_start_us":469,"threads_started":1}
I20260806 02:59:17.160181 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785985105255475-32043-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785985105255475-32043-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785985105255475-32043-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.195:0
--local_ip_for_outbound_sockets=127.31.74.195
--webserver_interface=127.31.74.195
--webserver_port=0
--tserver_master_addrs=127.31.74.254:41165
--builtin_ntp_servers=127.31.74.212:33911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260806 02:59:17.161252 4693 heartbeater.cc:347] Connected to a master server at 127.31.74.254:41165
I20260806 02:59:17.161643 4693 heartbeater.cc:464] Registering TS with master...
I20260806 02:59:17.162456 4693 heartbeater.cc:511] Master 127.31.74.254:41165 requested a full tablet report, sending...
I20260806 02:59:17.164410 4372 ts_manager.cc:194] Registered new tserver with Master: 82093717a7424668afbb0d6aac0c7596 (127.31.74.194:37741)
I20260806 02:59:17.569451 4560 heartbeater.cc:503] Master 127.31.74.254:41165 was elected leader, sending a full tablet report...
W20260806 02:59:17.589020 4697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:59:17.589375 4697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:59:17.589486 4697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:59:17.598387 4697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:59:17.598516 4697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:59:17.598664 4697 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.195
I20260806 02:59:17.610754 4697 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:33911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785985105255475-32043-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785985105255475-32043-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.74.195
--webserver_port=0
--tserver_master_addrs=127.31.74.254:41165
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.195
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785985105255475-32043-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:59:17.612578 4697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:59:17.617017 4697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:59:17.633749 4702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:17.633776 4705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:17.633713 4703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:59:17.635556 4697 server_base.cc:1034] running on GCE node
I20260806 02:59:17.636291 4697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:59:17.637570 4697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:59:17.638969 4697 hybrid_clock.cc:648] HybridClock initialized: now 1785985157638922 us; error 31 us; skew 500 ppm
I20260806 02:59:17.639320 4697 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 02:59:17.642308 4697 webserver.cc:460] Webserver started at http://127.31.74.195:33361/ using document root <none> and password file <none>
I20260806 02:59:17.643164 4697 fs_manager.cc:359] Metadata directory not provided
I20260806 02:59:17.643275 4697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:59:17.643589 4697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:59:17.647168 4697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785985105255475-32043-0/minicluster-data/ts-2/data/instance:
uuid: "d237a1833618469085cba254a6cf9332"
format_stamp: "Formatted at 2026-08-06 02:59:17 on dist-test-slave-dhph"
server_key: "76ea4fd7c8268badc81437f40d2dd6a3"
server_key_iv: "e03229cbdf4aeeacc2730af1d5a08f9b"
server_key_version: "encryptionkey@0"
I20260806 02:59:17.648218 4697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785985105255475-32043-0/minicluster-data/ts-2/wal/instance:
uuid: "d237a1833618469085cba254a6cf9332"
format_stamp: "Formatted at 2026-08-06 02:59:17 on dist-test-slave-dhph"
server_key: "76ea4fd7c8268badc81437f40d2dd6a3"
server_key_iv: "e03229cbdf4aeeacc2730af1d5a08f9b"
server_key_version: "encryptionkey@0"
I20260806 02:59:17.654909 4697 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260806 02:59:17.658818 4711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:17.660939 4697 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 02:59:17.661415 4697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785985105255475-32043-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785985105255475-32043-0/minicluster-data/ts-2/wal
uuid: "d237a1833618469085cba254a6cf9332"
format_stamp: "Formatted at 2026-08-06 02:59:17 on dist-test-slave-dhph"
server_key: "76ea4fd7c8268badc81437f40d2dd6a3"
server_key_iv: "e03229cbdf4aeeacc2730af1d5a08f9b"
server_key_version: "encryptionkey@0"
I20260806 02:59:17.661665 4697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785985105255475-32043-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785985105255475-32043-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785985105255475-32043-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:59:17.668140 4697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:59:17.669077 4697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:59:17.669374 4697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:59:17.670487 4697 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:59:17.672127 4697 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:59:17.672230 4697 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:17.672510 4697 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:59:17.672610 4697 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:17.714640 4697 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.195:36811
I20260806 02:59:17.714686 4827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.195:36811 every 8 connection(s)
I20260806 02:59:17.716138 4697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
I20260806 02:59:17.725507 4830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:17.719059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.195:46945 (local address 127.31.74.254:41165)
0806 02:59:17.719525 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:17.719530 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:17.720941 (+ 1411us) server_negotiation.cc:407] Connection header received
0806 02:59:17.722899 (+ 1958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:17.722905 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:17.722990 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:17.723121 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:17.724638 (+ 1517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:17.724650 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:17.724654 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:59:17.724672 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:17.724715 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:59:17.724727 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:17.724885 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:17.725078 (+ 193us) server_negotiation.cc:299] Negotiation successful
0806 02:59:17.725153 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":235,"threads_started":1}
I20260806 02:59:17.726019 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 4697
I20260806 02:59:17.726202 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785985105255475-32043-0/minicluster-data/ts-2/wal/instance
I20260806 02:59:17.726745 32043 external_mini_cluster.cc:1428] Setting key 5cc065fde20ca187e23e1dde2707fc89
I20260806 02:59:17.726854 4831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:17.719666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:41165 (local address 127.31.74.195:46945)
0806 02:59:17.720672 (+ 1006us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:17.720713 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:17.722536 (+ 1823us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:17.723387 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:17.723396 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:17.723497 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:59:17.724291 (+ 794us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:59:17.724295 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:59:17.724340 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:59:17.724425 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:59:17.724845 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:59:17.724856 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:59:17.724864 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 02:59:17.725168 (+ 304us) client_negotiation.cc:240] Negotiation successful
0806 02:59:17.725452 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":310,"threads_started":1}
I20260806 02:59:17.728700 4828 heartbeater.cc:347] Connected to a master server at 127.31.74.254:41165
I20260806 02:59:17.729092 4828 heartbeater.cc:464] Registering TS with master...
I20260806 02:59:17.729792 4828 heartbeater.cc:511] Master 127.31.74.254:41165 requested a full tablet report, sending...
I20260806 02:59:17.731549 4371 ts_manager.cc:194] Registered new tserver with Master: d237a1833618469085cba254a6cf9332 (127.31.74.195:36811)
I20260806 02:59:17.743340 32043 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:59:17.751286 4830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:17.748391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54966 (local address 127.31.74.254:41165)
0806 02:59:17.748599 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:17.748603 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:17.749244 (+ 641us) server_negotiation.cc:407] Connection header received
0806 02:59:17.749312 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:17.749316 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:17.749392 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:17.749484 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:17.750631 (+ 1147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:17.750639 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:17.750649 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:59:17.750668 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:17.750701 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:59:17.750754 (+ 53us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:17.750873 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:17.751011 (+ 138us) server_negotiation.cc:299] Negotiation successful
0806 02:59:17.751065 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260806 02:59:17.756402 4371 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54966:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:59:17.763821 4371 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:59:17.809878 4844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:17.777486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.195:36811 (local address 127.0.0.1:33878)
0806 02:59:17.802138 (+ 24652us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:17.802207 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:17.802481 (+ 274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:17.806030 (+ 3549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:17.806039 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:17.806142 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:59:17.807521 (+ 1379us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:59:17.807525 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:59:17.807573 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:59:17.807680 (+ 107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:59:17.808865 (+ 1185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:59:17.808878 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:59:17.808887 (+ 9us) client_negotiation.cc:769] Sending connection context
0806 02:59:17.809078 (+ 191us) client_negotiation.cc:240] Negotiation successful
0806 02:59:17.809174 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24241,"thread_start_us":27085,"threads_started":1}
I20260806 02:59:17.810883 4842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:17.777628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.194:37741 (local address 127.0.0.1:37440)
0806 02:59:17.804159 (+ 26531us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:17.804180 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:17.804303 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:17.806044 (+ 1741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:17.806048 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:17.806142 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:59:17.807959 (+ 1817us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:59:17.807962 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:59:17.807977 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:59:17.808026 (+ 49us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:59:17.810138 (+ 2112us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:59:17.810144 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:59:17.810150 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 02:59:17.810302 (+ 152us) client_negotiation.cc:240] Negotiation successful
0806 02:59:17.810360 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26359,"thread_start_us":26151,"threads_started":1}
I20260806 02:59:17.810969 4846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:17.778424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56070 (local address 127.31.74.193:41077)
0806 02:59:17.801065 (+ 22641us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:17.801082 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:17.805535 (+ 4453us) server_negotiation.cc:407] Connection header received
0806 02:59:17.805623 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:17.805631 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:17.805820 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:17.805991 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:17.809729 (+ 3738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:17.809742 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:17.809758 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:59:17.809789 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:17.809865 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:59:17.809894 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:17.810111 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:17.810413 (+ 302us) server_negotiation.cc:299] Negotiation successful
0806 02:59:17.810540 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22347,"thread_start_us":22671,"threads_started":1}
I20260806 02:59:17.811069 4843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:17.777654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33878 (local address 127.31.74.195:36811)
0806 02:59:17.805273 (+ 27619us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:17.805284 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:17.805327 (+ 43us) server_negotiation.cc:407] Connection header received
0806 02:59:17.805409 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:17.805421 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:17.805719 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:17.806958 (+ 1239us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:17.808214 (+ 1256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:17.808221 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:17.808238 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:59:17.808259 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:17.808300 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:59:17.808314 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:17.808503 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:17.810472 (+ 1969us) server_negotiation.cc:299] Negotiation successful
0806 02:59:17.810569 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27411,"thread_start_us":27237,"threads_started":1}
I20260806 02:59:17.812155 4845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:17.778318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37440 (local address 127.31.74.194:37741)
0806 02:59:17.805392 (+ 27074us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:17.805399 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:17.805430 (+ 31us) server_negotiation.cc:407] Connection header received
0806 02:59:17.805528 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:17.805537 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:17.805781 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:17.806051 (+ 270us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:17.809837 (+ 3786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:17.809846 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:17.809859 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:59:17.809885 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:17.809933 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:59:17.809950 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:17.810082 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:17.811697 (+ 1615us) server_negotiation.cc:299] Negotiation successful
0806 02:59:17.811805 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26868,"thread_start_us":27282,"threads_started":1}
I20260806 02:59:17.812422 4841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:17.777904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:41077 (local address 127.0.0.1:56070)
0806 02:59:17.805385 (+ 27481us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:17.805407 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:17.805524 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:17.808455 (+ 2931us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:17.808464 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:17.808492 (+ 28us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:59:17.809293 (+ 801us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:59:17.809297 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:59:17.809319 (+ 22us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:59:17.809367 (+ 48us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:59:17.810174 (+ 807us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:59:17.810179 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:59:17.810185 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 02:59:17.811349 (+ 1164us) client_negotiation.cc:240] Negotiation successful
0806 02:59:17.811398 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27342,"thread_start_us":24896,"threads_started":1}
I20260806 02:59:17.817342 4627 tablet_service.cc:1467] Processing CreateTablet for tablet 61d8f3e22f994b3b95b4e03d75266e6a (DEFAULT_TABLE table=test-table [id=677cddbe42ba47bdb8e0cf9c866f7e8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:59:17.817548 4762 tablet_service.cc:1467] Processing CreateTablet for tablet 61d8f3e22f994b3b95b4e03d75266e6a (DEFAULT_TABLE table=test-table [id=677cddbe42ba47bdb8e0cf9c866f7e8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:59:17.817819 4494 tablet_service.cc:1467] Processing CreateTablet for tablet 61d8f3e22f994b3b95b4e03d75266e6a (DEFAULT_TABLE table=test-table [id=677cddbe42ba47bdb8e0cf9c866f7e8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:59:17.819152 4762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61d8f3e22f994b3b95b4e03d75266e6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:59:17.819289 4494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61d8f3e22f994b3b95b4e03d75266e6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:59:17.819427 4627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61d8f3e22f994b3b95b4e03d75266e6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:59:17.835867 4847 tablet_bootstrap.cc:492] T 61d8f3e22f994b3b95b4e03d75266e6a P d237a1833618469085cba254a6cf9332: Bootstrap starting.
I20260806 02:59:17.838361 4848 tablet_bootstrap.cc:492] T 61d8f3e22f994b3b95b4e03d75266e6a P ccc5e8406a4d45349fbc6ca8d4a65108: Bootstrap starting.
I20260806 02:59:17.840265 4849 tablet_bootstrap.cc:492] T 61d8f3e22f994b3b95b4e03d75266e6a P 82093717a7424668afbb0d6aac0c7596: Bootstrap starting.
I20260806 02:59:17.842559 4847 tablet_bootstrap.cc:654] T 61d8f3e22f994b3b95b4e03d75266e6a P d237a1833618469085cba254a6cf9332: Neither blocks nor log segments found. Creating new log.
I20260806 02:59:17.843050 4848 tablet_bootstrap.cc:654] T 61d8f3e22f994b3b95b4e03d75266e6a P ccc5e8406a4d45349fbc6ca8d4a65108: Neither blocks nor log segments found. Creating new log.
I20260806 02:59:17.843329 4849 tablet_bootstrap.cc:654] T 61d8f3e22f994b3b95b4e03d75266e6a P 82093717a7424668afbb0d6aac0c7596: Neither blocks nor log segments found. Creating new log.
I20260806 02:59:17.844170 4848 log.cc:824] T 61d8f3e22f994b3b95b4e03d75266e6a P ccc5e8406a4d45349fbc6ca8d4a65108: Log is configured to *not* fsync() on all Append() calls
I20260806 02:59:17.844154 4847 log.cc:824] T 61d8f3e22f994b3b95b4e03d75266e6a P d237a1833618469085cba254a6cf9332: Log is configured to *not* fsync() on all Append() calls
I20260806 02:59:17.844328 4849 log.cc:824] T 61d8f3e22f994b3b95b4e03d75266e6a P 82093717a7424668afbb0d6aac0c7596: Log is configured to *not* fsync() on all Append() calls
I20260806 02:59:17.847882 4848 tablet_bootstrap.cc:492] T 61d8f3e22f994b3b95b4e03d75266e6a P ccc5e8406a4d45349fbc6ca8d4a65108: No bootstrap required, opened a new log
I20260806 02:59:17.847882 4847 tablet_bootstrap.cc:492] T 61d8f3e22f994b3b95b4e03d75266e6a P d237a1833618469085cba254a6cf9332: No bootstrap required, opened a new log
I20260806 02:59:17.848201 4848 ts_tablet_manager.cc:1397] T 61d8f3e22f994b3b95b4e03d75266e6a P ccc5e8406a4d45349fbc6ca8d4a65108: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260806 02:59:17.848201 4847 ts_tablet_manager.cc:1397] T 61d8f3e22f994b3b95b4e03d75266e6a P d237a1833618469085cba254a6cf9332: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.004s
I20260806 02:59:17.848742 4849 tablet_bootstrap.cc:492] T 61d8f3e22f994b3b95b4e03d75266e6a P 82093717a7424668afbb0d6aac0c7596: No bootstrap required, opened a new log
I20260806 02:59:17.849016 4849 ts_tablet_manager.cc:1397] T 61d8f3e22f994b3b95b4e03d75266e6a P 82093717a7424668afbb0d6aac0c7596: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260806 02:59:17.859524 4849 raft_consensus.cc:348] T 61d8f3e22f994b3b95b4e03d75266e6a P 82093717a7424668afbb0d6aac0c7596 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d237a1833618469085cba254a6cf9332" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 36811 } } peers { permanent_uuid: "82093717a7424668afbb0d6aac0c7596" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 37741 } } peers { permanent_uuid: "ccc5e8406a4d45349fbc6ca8d4a65108" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 41077 } }
I20260806 02:59:17.860149 4849 raft_consensus.cc:374] T 61d8f3e22f994b3b95b4e03d75266e6a P 82093717a7424668afbb0d6aac0c7596 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:59:17.860301 4849 raft_consensus.cc:729] T 61d8f3e22f994b3b95b4e03d75266e6a P 82093717a7424668afbb0d6aac0c7596 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82093717a7424668afbb0d6aac0c7596, State: Initialized, Role: FOLLOWER
I20260806 02:59:17.861030 4849 consensus_queue.cc:260] T 61d8f3e22f994b3b95b4e03d75266e6a P 82093717a7424668afbb0d6aac0c7596 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d237a1833618469085cba254a6cf9332" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 36811 } } peers { permanent_uuid: "82093717a7424668afbb0d6aac0c7596" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 37741 } } peers { permanent_uuid: "ccc5e8406a4d45349fbc6ca8d4a65108" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 41077 } }
I20260806 02:59:17.860903 4847 raft_consensus.cc:348] T 61d8f3e22f994b3b95b4e03d75266e6a P d237a1833618469085cba254a6cf9332 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d237a1833618469085cba254a6cf9332" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 36811 } } peers { permanent_uuid: "82093717a7424668afbb0d6aac0c7596" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 37741 } } peers { permanent_uuid: "ccc5e8406a4d45349fbc6ca8d4a65108" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 41077 } }
I20260806 02:59:17.860904 4848 raft_consensus.cc:348] T 61d8f3e22f994b3b95b4e03d75266e6a P ccc5e8406a4d45349fbc6ca8d4a65108 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d237a1833618469085cba254a6cf9332" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 36811 } } peers { permanent_uuid: "82093717a7424668afbb0d6aac0c7596" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 37741 } } peers { permanent_uuid: "ccc5e8406a4d45349fbc6ca8d4a65108" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 41077 } }
I20260806 02:59:17.861405 4848 raft_consensus.cc:374] T 61d8f3e22f994b3b95b4e03d75266e6a P ccc5e8406a4d45349fbc6ca8d4a65108 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:59:17.861444 4847 raft_consensus.cc:374] T 61d8f3e22f994b3b95b4e03d75266e6a P d237a1833618469085cba254a6cf9332 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:59:17.861545 4848 raft_consensus.cc:729] T 61d8f3e22f994b3b95b4e03d75266e6a P ccc5e8406a4d45349fbc6ca8d4a65108 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccc5e8406a4d45349fbc6ca8d4a65108, State: Initialized, Role: FOLLOWER
I20260806 02:59:17.861545 4847 raft_consensus.cc:729] T 61d8f3e22f994b3b95b4e03d75266e6a P d237a1833618469085cba254a6cf9332 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d237a1833618469085cba254a6cf9332, State: Initialized, Role: FOLLOWER
I20260806 02:59:17.862344 4693 heartbeater.cc:503] Master 127.31.74.254:41165 was elected leader, sending a full tablet report...
I20260806 02:59:17.862537 4848 consensus_queue.cc:260] T 61d8f3e22f994b3b95b4e03d75266e6a P ccc5e8406a4d45349fbc6ca8d4a65108 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d237a1833618469085cba254a6cf9332" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 36811 } } peers { permanent_uuid: "82093717a7424668afbb0d6aac0c7596" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 37741 } } peers { permanent_uuid: "ccc5e8406a4d45349fbc6ca8d4a65108" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 41077 } }
I20260806 02:59:17.863075 4849 ts_tablet_manager.cc:1428] T 61d8f3e22f994b3b95b4e03d75266e6a P 82093717a7424668afbb0d6aac0c7596: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260806 02:59:17.863559 4847 consensus_queue.cc:260] T 61d8f3e22f994b3b95b4e03d75266e6a P d237a1833618469085cba254a6cf9332 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d237a1833618469085cba254a6cf9332" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 36811 } } peers { permanent_uuid: "82093717a7424668afbb0d6aac0c7596" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 37741 } } peers { permanent_uuid: "ccc5e8406a4d45349fbc6ca8d4a65108" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 41077 } }
I20260806 02:59:17.864723 4848 ts_tablet_manager.cc:1428] T 61d8f3e22f994b3b95b4e03d75266e6a P ccc5e8406a4d45349fbc6ca8d4a65108: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20260806 02:59:17.869439 4828 heartbeater.cc:503] Master 127.31.74.254:41165 was elected leader, sending a full tablet report...
I20260806 02:59:17.871098 4847 ts_tablet_manager.cc:1428] T 61d8f3e22f994b3b95b4e03d75266e6a P d237a1833618469085cba254a6cf9332: Time spent starting tablet: real 0.023s user 0.016s sys 0.005s
I20260806 02:59:17.871757 4853 raft_consensus.cc:482] T 61d8f3e22f994b3b95b4e03d75266e6a P 82093717a7424668afbb0d6aac0c7596 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:59:17.872098 4853 raft_consensus.cc:504] T 61d8f3e22f994b3b95b4e03d75266e6a P 82093717a7424668afbb0d6aac0c7596 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d237a1833618469085cba254a6cf9332" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 36811 } } peers { permanent_uuid: "82093717a7424668afbb0d6aac0c7596" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 37741 } } peers { permanent_uuid: "ccc5e8406a4d45349fbc6ca8d4a65108" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 41077 } }
I20260806 02:59:17.876015 4853 leader_election.cc:288] T 61d8f3e22f994b3b95b4e03d75266e6a P 82093717a7424668afbb0d6aac0c7596 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d237a1833618469085cba254a6cf9332 (127.31.74.195:36811), ccc5e8406a4d45349fbc6ca8d4a65108 (127.31.74.193:41077)
I20260806 02:59:17.880427 4846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:17.875820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.194:39595 (local address 127.31.74.193:41077)
0806 02:59:17.876028 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:17.876034 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:17.877754 (+ 1720us) server_negotiation.cc:407] Connection header received
0806 02:59:17.877923 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:17.877928 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:17.878063 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:17.878299 (+ 236us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:17.879687 (+ 1388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:17.879693 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:17.879697 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:59:17.879716 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:17.879750 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:59:17.879763 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:17.879880 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:17.880026 (+ 146us) server_negotiation.cc:299] Negotiation successful
0806 02:59:17.880107 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 02:59:17.880479 4843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:17.876424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.194:60861 (local address 127.31.74.195:36811)
0806 02:59:17.876615 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:17.876624 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:17.877193 (+ 569us) server_negotiation.cc:407] Connection header received
0806 02:59:17.877455 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:17.877460 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:17.877513 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:17.877637 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:17.879411 (+ 1774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:17.879418 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:17.879423 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:59:17.879447 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:17.879490 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:59:17.879510 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:17.879733 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:17.880085 (+ 352us) server_negotiation.cc:299] Negotiation successful
0806 02:59:17.880166 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260806 02:59:17.880687 4856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:17.875624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.195:36811 (local address 127.31.74.194:60861)
0806 02:59:17.877064 (+ 1440us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:17.877089 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:17.877254 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:17.877668 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:17.877673 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:17.877703 (+ 30us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:59:17.878945 (+ 1242us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:59:17.878948 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:59:17.878991 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:59:17.879060 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:59:17.879932 (+ 872us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:59:17.879936 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:59:17.879940 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 02:59:17.880126 (+ 186us) client_negotiation.cc:240] Negotiation successful
0806 02:59:17.880187 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1259,"thread_start_us":1059,"threads_started":1}
I20260806 02:59:17.881223 4857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:17.875802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:41077 (local address 127.31.74.194:39595)
0806 02:59:17.877626 (+ 1824us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:17.877643 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:17.877797 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:17.878361 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:17.878366 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:17.878393 (+ 27us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:59:17.879446 (+ 1053us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:59:17.879449 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:59:17.879467 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:59:17.879512 (+ 45us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:59:17.879873 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:59:17.879876 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:59:17.879880 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 02:59:17.880075 (+ 195us) client_negotiation.cc:240] Negotiation successful
0806 02:59:17.880149 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1689,"thread_start_us":1455,"threads_started":1}
I20260806 02:59:17.881822 4782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61d8f3e22f994b3b95b4e03d75266e6a" candidate_uuid: "82093717a7424668afbb0d6aac0c7596" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d237a1833618469085cba254a6cf9332" is_pre_election: true
I20260806 02:59:17.882352 4782 raft_consensus.cc:2454] T 61d8f3e22f994b3b95b4e03d75266e6a P d237a1833618469085cba254a6cf9332 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82093717a7424668afbb0d6aac0c7596 in term 0.
I20260806 02:59:17.883483 4583 leader_election.cc:302] T 61d8f3e22f994b3b95b4e03d75266e6a P 82093717a7424668afbb0d6aac0c7596 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82093717a7424668afbb0d6aac0c7596, d237a1833618469085cba254a6cf9332; no voters:
I20260806 02:59:17.883946 4853 raft_consensus.cc:2781] T 61d8f3e22f994b3b95b4e03d75266e6a P 82093717a7424668afbb0d6aac0c7596 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:59:17.884119 4853 raft_consensus.cc:482] T 61d8f3e22f994b3b95b4e03d75266e6a P 82093717a7424668afbb0d6aac0c7596 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:59:17.884255 4853 raft_consensus.cc:3037] T 61d8f3e22f994b3b95b4e03d75266e6a P 82093717a7424668afbb0d6aac0c7596 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:59:17.884311 4514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61d8f3e22f994b3b95b4e03d75266e6a" candidate_uuid: "82093717a7424668afbb0d6aac0c7596" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccc5e8406a4d45349fbc6ca8d4a65108" is_pre_election: true
I20260806 02:59:17.885573 4514 raft_consensus.cc:2454] T 61d8f3e22f994b3b95b4e03d75266e6a P ccc5e8406a4d45349fbc6ca8d4a65108 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82093717a7424668afbb0d6aac0c7596 in term 0.
I20260806 02:59:17.887766 4853 raft_consensus.cc:504] T 61d8f3e22f994b3b95b4e03d75266e6a P 82093717a7424668afbb0d6aac0c7596 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d237a1833618469085cba254a6cf9332" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 36811 } } peers { permanent_uuid: "82093717a7424668afbb0d6aac0c7596" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 37741 } } peers { permanent_uuid: "ccc5e8406a4d45349fbc6ca8d4a65108" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 41077 } }
I20260806 02:59:17.889253 4853 leader_election.cc:288] T 61d8f3e22f994b3b95b4e03d75266e6a P 82093717a7424668afbb0d6aac0c7596 [CANDIDATE]: Term 1 election: Requested vote from peers d237a1833618469085cba254a6cf9332 (127.31.74.195:36811), ccc5e8406a4d45349fbc6ca8d4a65108 (127.31.74.193:41077)
I20260806 02:59:17.890107 4782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61d8f3e22f994b3b95b4e03d75266e6a" candidate_uuid: "82093717a7424668afbb0d6aac0c7596" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d237a1833618469085cba254a6cf9332"
I20260806 02:59:17.890008 4514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61d8f3e22f994b3b95b4e03d75266e6a" candidate_uuid: "82093717a7424668afbb0d6aac0c7596" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccc5e8406a4d45349fbc6ca8d4a65108"
I20260806 02:59:17.890475 4514 raft_consensus.cc:3037] T 61d8f3e22f994b3b95b4e03d75266e6a P ccc5e8406a4d45349fbc6ca8d4a65108 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:59:17.890477 4782 raft_consensus.cc:3037] T 61d8f3e22f994b3b95b4e03d75266e6a P d237a1833618469085cba254a6cf9332 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:59:17.894646 4514 raft_consensus.cc:2454] T 61d8f3e22f994b3b95b4e03d75266e6a P ccc5e8406a4d45349fbc6ca8d4a65108 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82093717a7424668afbb0d6aac0c7596 in term 1.
I20260806 02:59:17.894640 4782 raft_consensus.cc:2454] T 61d8f3e22f994b3b95b4e03d75266e6a P d237a1833618469085cba254a6cf9332 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82093717a7424668afbb0d6aac0c7596 in term 1.
I20260806 02:59:17.895242 4582 leader_election.cc:302] T 61d8f3e22f994b3b95b4e03d75266e6a P 82093717a7424668afbb0d6aac0c7596 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82093717a7424668afbb0d6aac0c7596, ccc5e8406a4d45349fbc6ca8d4a65108; no voters:
I20260806 02:59:17.895635 4853 raft_consensus.cc:2781] T 61d8f3e22f994b3b95b4e03d75266e6a P 82093717a7424668afbb0d6aac0c7596 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:59:17.896224 4853 raft_consensus.cc:686] T 61d8f3e22f994b3b95b4e03d75266e6a P 82093717a7424668afbb0d6aac0c7596 [term 1 LEADER]: Becoming Leader. State: Replica: 82093717a7424668afbb0d6aac0c7596, State: Running, Role: LEADER
I20260806 02:59:17.896828 4853 consensus_queue.cc:237] T 61d8f3e22f994b3b95b4e03d75266e6a P 82093717a7424668afbb0d6aac0c7596 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d237a1833618469085cba254a6cf9332" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 36811 } } peers { permanent_uuid: "82093717a7424668afbb0d6aac0c7596" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 37741 } } peers { permanent_uuid: "ccc5e8406a4d45349fbc6ca8d4a65108" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 41077 } }
W20260806 02:59:17.899048 4694 tablet.cc:2369] T 61d8f3e22f994b3b95b4e03d75266e6a P 82093717a7424668afbb0d6aac0c7596: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:59:17.903162 4372 catalog_manager.cc:5495] T 61d8f3e22f994b3b95b4e03d75266e6a P 82093717a7424668afbb0d6aac0c7596 reported cstate change: term changed from 0 to 1, leader changed from <none> to 82093717a7424668afbb0d6aac0c7596 (127.31.74.194). New cstate: current_term: 1 leader_uuid: "82093717a7424668afbb0d6aac0c7596" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d237a1833618469085cba254a6cf9332" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 36811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82093717a7424668afbb0d6aac0c7596" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 37741 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ccc5e8406a4d45349fbc6ca8d4a65108" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 41077 } health_report { overall_health: UNKNOWN } } }
I20260806 02:59:17.937264 4845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:17.934665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37456 (local address 127.31.74.194:37741)
0806 02:59:17.934964 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:17.934984 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:17.935009 (+ 25us) server_negotiation.cc:407] Connection header received
0806 02:59:17.935301 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:17.935306 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:17.935400 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:17.935528 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:17.936559 (+ 1031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:17.936571 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:17.936575 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:59:17.936593 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:17.936627 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:59:17.936642 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:17.936790 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:17.936876 (+ 86us) server_negotiation.cc:299] Negotiation successful
0806 02:59:17.936946 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146}
I20260806 02:59:17.948357 4782 raft_consensus.cc:1261] T 61d8f3e22f994b3b95b4e03d75266e6a P d237a1833618469085cba254a6cf9332 [term 1 FOLLOWER]: Refusing update from remote peer 82093717a7424668afbb0d6aac0c7596: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:59:17.948585 4514 raft_consensus.cc:1261] T 61d8f3e22f994b3b95b4e03d75266e6a P ccc5e8406a4d45349fbc6ca8d4a65108 [term 1 FOLLOWER]: Refusing update from remote peer 82093717a7424668afbb0d6aac0c7596: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:59:17.949672 4858 consensus_queue.cc:1035] T 61d8f3e22f994b3b95b4e03d75266e6a P 82093717a7424668afbb0d6aac0c7596 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d237a1833618469085cba254a6cf9332" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 36811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:59:17.950119 4853 consensus_queue.cc:1035] T 61d8f3e22f994b3b95b4e03d75266e6a P 82093717a7424668afbb0d6aac0c7596 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccc5e8406a4d45349fbc6ca8d4a65108" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 41077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:59:17.965734 4861 mvcc.cc:205] Tried to move back new op lower bound from 7315395206942789632 to 7315395206751268864. Current Snapshot: MvccSnapshot[applied={T|T < 7315395206942789632}]
I20260806 02:59:17.986076 4372 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54966:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:59:17.986537 4372 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54966:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:59:17.992209 4372 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 54cb2d1b3cb0430fbe9a6fccc870fbb3: Sending DeleteTablet for 3 replicas of tablet 61d8f3e22f994b3b95b4e03d75266e6a
I20260806 02:59:17.994047 4627 tablet_service.cc:1514] Processing DeleteTablet for tablet 61d8f3e22f994b3b95b4e03d75266e6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:59:17 UTC) from {username='slave'} at 127.0.0.1:37440
I20260806 02:59:17.994047 4762 tablet_service.cc:1514] Processing DeleteTablet for tablet 61d8f3e22f994b3b95b4e03d75266e6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:59:17 UTC) from {username='slave'} at 127.0.0.1:33878
I20260806 02:59:17.994041 4494 tablet_service.cc:1514] Processing DeleteTablet for tablet 61d8f3e22f994b3b95b4e03d75266e6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:59:17 UTC) from {username='slave'} at 127.0.0.1:56070
I20260806 02:59:17.999015 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 4432
I20260806 02:59:18.009538 4846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:18.002633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56086 (local address 127.31.74.193:41077)
0806 02:59:18.002953 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:18.002960 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:18.003219 (+ 259us) server_negotiation.cc:407] Connection header received
0806 02:59:18.005129 (+ 1910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:18.005140 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:18.005326 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:18.005428 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:18.007709 (+ 2281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:18.007720 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:18.007729 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:59:18.007765 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:18.007827 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:59:18.007852 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:18.008020 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:18.008589 (+ 569us) server_negotiation.cc:299] Negotiation successful
0806 02:59:18.008693 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137}
I20260806 02:59:18.012804 4555 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:59:18.017030 4871 tablet_replica.cc:330] T 61d8f3e22f994b3b95b4e03d75266e6a P 82093717a7424668afbb0d6aac0c7596: stopping tablet replica
I20260806 02:59:18.018851 4871 raft_consensus.cc:2229] T 61d8f3e22f994b3b95b4e03d75266e6a P 82093717a7424668afbb0d6aac0c7596 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:59:18.020936 4871 raft_consensus.cc:2258] T 61d8f3e22f994b3b95b4e03d75266e6a P 82093717a7424668afbb0d6aac0c7596 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:59:18.026566 4871 ts_tablet_manager.cc:1905] T 61d8f3e22f994b3b95b4e03d75266e6a P 82093717a7424668afbb0d6aac0c7596: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:59:18.028285 4870 tablet_replica.cc:330] T 61d8f3e22f994b3b95b4e03d75266e6a P d237a1833618469085cba254a6cf9332: stopping tablet replica
I20260806 02:59:18.030157 4870 raft_consensus.cc:2229] T 61d8f3e22f994b3b95b4e03d75266e6a P d237a1833618469085cba254a6cf9332 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:59:18.031625 4870 raft_consensus.cc:2258] T 61d8f3e22f994b3b95b4e03d75266e6a P d237a1833618469085cba254a6cf9332 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:59:18.036814 4870 ts_tablet_manager.cc:1905] T 61d8f3e22f994b3b95b4e03d75266e6a P d237a1833618469085cba254a6cf9332: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:59:18.039081 4871 ts_tablet_manager.cc:1918] T 61d8f3e22f994b3b95b4e03d75266e6a P 82093717a7424668afbb0d6aac0c7596: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:59:18.039377 4871 log.cc:1196] T 61d8f3e22f994b3b95b4e03d75266e6a P 82093717a7424668afbb0d6aac0c7596: Deleting WAL directory at /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785985105255475-32043-0/minicluster-data/ts-1/wal/wals/61d8f3e22f994b3b95b4e03d75266e6a
I20260806 02:59:18.040198 4871 ts_tablet_manager.cc:1939] T 61d8f3e22f994b3b95b4e03d75266e6a P 82093717a7424668afbb0d6aac0c7596: Deleting consensus metadata
I20260806 02:59:18.043052 4356 catalog_manager.cc:4841] TS 82093717a7424668afbb0d6aac0c7596 (127.31.74.194:37741): tablet 61d8f3e22f994b3b95b4e03d75266e6a (table test-table [id=677cddbe42ba47bdb8e0cf9c866f7e8b]) successfully deleted
I20260806 02:59:18.044510 4870 ts_tablet_manager.cc:1918] T 61d8f3e22f994b3b95b4e03d75266e6a P d237a1833618469085cba254a6cf9332: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:59:18.044719 4870 log.cc:1196] T 61d8f3e22f994b3b95b4e03d75266e6a P d237a1833618469085cba254a6cf9332: Deleting WAL directory at /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785985105255475-32043-0/minicluster-data/ts-2/wal/wals/61d8f3e22f994b3b95b4e03d75266e6a
I20260806 02:59:18.045502 4870 ts_tablet_manager.cc:1939] T 61d8f3e22f994b3b95b4e03d75266e6a P d237a1833618469085cba254a6cf9332: Deleting consensus metadata
I20260806 02:59:18.047705 4359 catalog_manager.cc:4841] TS d237a1833618469085cba254a6cf9332 (127.31.74.195:36811): tablet 61d8f3e22f994b3b95b4e03d75266e6a (table test-table [id=677cddbe42ba47bdb8e0cf9c866f7e8b]) successfully deleted
I20260806 02:59:18.159394 4872 tablet_replica.cc:330] T 61d8f3e22f994b3b95b4e03d75266e6a P ccc5e8406a4d45349fbc6ca8d4a65108: stopping tablet replica
I20260806 02:59:18.159515 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 4432
I20260806 02:59:18.159932 4872 raft_consensus.cc:2229] T 61d8f3e22f994b3b95b4e03d75266e6a P ccc5e8406a4d45349fbc6ca8d4a65108 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:59:18.195518 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 4564
W20260806 02:59:18.195537 4358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.74.193:41077 (error 108)
I20260806 02:59:18.199522 4841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:18.198847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:41077 (local address 127.0.0.1:56094)
0806 02:59:18.199154 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:18.199318 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.193:41077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160}
I20260806 02:59:18.200423 4845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:18.197451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37466 (local address 127.31.74.194:37741)
0806 02:59:18.197660 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:18.197668 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:18.197691 (+ 23us) server_negotiation.cc:407] Connection header received
0806 02:59:18.197809 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:18.197816 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:18.197888 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:18.198022 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:18.199306 (+ 1284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:18.199311 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:18.199317 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:59:18.199338 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:18.199379 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:59:18.199395 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:18.199514 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:18.200030 (+ 516us) server_negotiation.cc:299] Negotiation successful
0806 02:59:18.200099 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
W20260806 02:59:18.200379 4358 catalog_manager.cc:4584] TS ccc5e8406a4d45349fbc6ca8d4a65108 (127.31.74.193:41077): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 61d8f3e22f994b3b95b4e03d75266e6a: Network error: Client connection negotiation failed: client connection to 127.31.74.193:41077: connect: Connection refused (error 111)
I20260806 02:59:18.201505 4688 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:59:18.254724 4841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:18.254051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:41077 (local address 127.0.0.1:56104)
0806 02:59:18.254342 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:18.254471 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.193:41077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260806 02:59:18.256093 4841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:18.255649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:41077 (local address 127.0.0.1:56106)
0806 02:59:18.255839 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:18.255934 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.193:41077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260806 02:59:18.316973 4841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:18.316387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:41077 (local address 127.0.0.1:56112)
0806 02:59:18.316655 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:18.316777 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.193:41077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260806 02:59:18.318224 4841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:18.317798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:41077 (local address 127.0.0.1:56114)
0806 02:59:18.317969 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:18.318063 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.193:41077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260806 02:59:18.328593 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 4564
I20260806 02:59:18.352741 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 4697
I20260806 02:59:18.356722 4843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:18.353762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33890 (local address 127.31.74.195:36811)
0806 02:59:18.353959 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:18.353963 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:18.354181 (+ 218us) server_negotiation.cc:407] Connection header received
0806 02:59:18.354257 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:18.354261 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:18.354777 (+ 516us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:18.354896 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:18.356018 (+ 1122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:18.356022 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:18.356026 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:59:18.356045 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:18.356083 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:59:18.356099 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:18.356230 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:18.356399 (+ 169us) server_negotiation.cc:299] Negotiation successful
0806 02:59:18.356458 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260806 02:59:18.357410 4822 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:59:18.399343 4841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:18.398704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:41077 (local address 127.0.0.1:56124)
0806 02:59:18.399017 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:18.399148 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.193:41077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161}
I20260806 02:59:18.400791 4841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:18.400331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:41077 (local address 127.0.0.1:56130)
0806 02:59:18.400560 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:18.400658 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.193:41077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260806 02:59:18.462800 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 4697
I20260806 02:59:18.486507 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 4341
I20260806 02:59:18.488240 4402 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:59:18.619500 4841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:18.618998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:41077 (local address 127.0.0.1:56132)
0806 02:59:18.619239 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:18.619340 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.193:41077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260806 02:59:18.620229 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 4341
2026-08-06T02:59:18Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (3399 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260806 02:59:18.671782 32043 mini_oidc.cc:152] Starting JWKS server
I20260806 02:59:18.675207 32043 webserver.cc:460] Webserver started at https://127.0.0.1:41489/ using document root <none> and password file <none>
I20260806 02:59:18.675642 32043 mini_oidc.cc:189] Starting OIDC Discovery server
I20260806 02:59:18.676429 32043 webserver.cc:460] Webserver started at http://127.0.0.1:41373/ using document root <none> and password file <none>
2026-08-06T02:59:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:59:18Z Disabled control of system clock
I20260806 02:59:18.723032 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785985105255475-32043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785985105255475-32043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785985105255475-32043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:44127
--webserver_interface=127.31.74.254
--webserver_port=0
--builtin_ntp_servers=127.31.74.212:37379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.74.254:44127
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41489/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785985105255475-32043-0/testchainca.cert with env {}
W20260806 02:59:19.120150 4884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:59:19.120532 4884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:59:19.120710 4884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:59:19.128327 4884 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:59:19.128511 4884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:59:19.128613 4884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:59:19.128728 4884 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:59:19.128819 4884 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 02:59:19.128913 4884 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 02:59:19.128993 4884 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41489/jwks.json
I20260806 02:59:19.140580 4884 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:37379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785985105255475-32043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785985105255475-32043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.74.254:44127
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:44127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41489/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.74.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785985105255475-32043-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785985105255475-32043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:59:19.142083 4884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:59:19.143554 4884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:59:19.160291 4890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:19.160671 4892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:19.161397 4889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:59:19.163722 4884 server_base.cc:1034] running on GCE node
I20260806 02:59:19.164759 4884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:59:19.166428 4884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:59:19.167716 4884 hybrid_clock.cc:648] HybridClock initialized: now 1785985159167673 us; error 67 us; skew 500 ppm
I20260806 02:59:19.168318 4884 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 02:59:19.172078 4884 webserver.cc:460] Webserver started at http://127.31.74.254:42689/ using document root <none> and password file <none>
I20260806 02:59:19.173013 4884 fs_manager.cc:359] Metadata directory not provided
I20260806 02:59:19.173177 4884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:59:19.173548 4884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:59:19.177289 4884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785985105255475-32043-0/minicluster-data/master-0/data/instance:
uuid: "981fa569ac49410ea40740b68baa95fa"
format_stamp: "Formatted at 2026-08-06 02:59:19 on dist-test-slave-dhph"
server_key: "2082487f3c4667a5a2aa6dc561e3beb1"
server_key_iv: "f37c5eb4c66880f075d3ebe545bb2762"
server_key_version: "encryptionkey@0"
I20260806 02:59:19.178293 4884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785985105255475-32043-0/minicluster-data/master-0/wal/instance:
uuid: "981fa569ac49410ea40740b68baa95fa"
format_stamp: "Formatted at 2026-08-06 02:59:19 on dist-test-slave-dhph"
server_key: "2082487f3c4667a5a2aa6dc561e3beb1"
server_key_iv: "f37c5eb4c66880f075d3ebe545bb2762"
server_key_version: "encryptionkey@0"
I20260806 02:59:19.184341 4884 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260806 02:59:19.188519 4898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:19.190994 4884 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 02:59:19.191303 4884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785985105255475-32043-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785985105255475-32043-0/minicluster-data/master-0/wal
uuid: "981fa569ac49410ea40740b68baa95fa"
format_stamp: "Formatted at 2026-08-06 02:59:19 on dist-test-slave-dhph"
server_key: "2082487f3c4667a5a2aa6dc561e3beb1"
server_key_iv: "f37c5eb4c66880f075d3ebe545bb2762"
server_key_version: "encryptionkey@0"
I20260806 02:59:19.191641 4884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785985105255475-32043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785985105255475-32043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785985105255475-32043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:59:19.218968 4884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:59:19.219921 4884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:59:19.220269 4884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:59:19.244447 4884 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.254:44127
I20260806 02:59:19.244472 4950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.254:44127 every 8 connection(s)
I20260806 02:59:19.245956 4884 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
I20260806 02:59:19.250094 4951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:59:19.254873 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 4884
I20260806 02:59:19.255117 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785985105255475-32043-0/minicluster-data/master-0/wal/instance
I20260806 02:59:19.255762 32043 external_mini_cluster.cc:1428] Setting key 0aa86255166c4d8f888047ef4bc9949b
I20260806 02:59:19.263841 4951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 981fa569ac49410ea40740b68baa95fa: Bootstrap starting.
I20260806 02:59:19.268466 4951 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 981fa569ac49410ea40740b68baa95fa: Neither blocks nor log segments found. Creating new log.
I20260806 02:59:19.269819 4951 log.cc:824] T 00000000000000000000000000000000 P 981fa569ac49410ea40740b68baa95fa: Log is configured to *not* fsync() on all Append() calls
I20260806 02:59:19.273936 4951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 981fa569ac49410ea40740b68baa95fa: No bootstrap required, opened a new log
I20260806 02:59:19.274982 4954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:19.258006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46868 (local address 127.31.74.254:44127)
0806 02:59:19.259304 (+ 1298us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:19.259328 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:19.259394 (+ 66us) server_negotiation.cc:407] Connection header received
0806 02:59:19.262221 (+ 2827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:19.262255 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:19.262970 (+ 715us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:19.263494 (+ 524us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:19.264229 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:19.266496 (+ 2267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:19.268896 (+ 2400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:19.270070 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:19.270335 (+ 265us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:19.271988 (+ 1653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:19.271998 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:19.272017 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:59:19.272740 (+ 723us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:19.272849 (+ 109us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:59:19.272876 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:19.273024 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:19.273316 (+ 292us) server_negotiation.cc:299] Negotiation successful
0806 02:59:19.273644 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":725,"thread_start_us":407,"threads_started":1}
I20260806 02:59:19.279752 32043 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:59:19.284907 4951 raft_consensus.cc:348] T 00000000000000000000000000000000 P 981fa569ac49410ea40740b68baa95fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "981fa569ac49410ea40740b68baa95fa" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 44127 } }
I20260806 02:59:19.285316 4951 raft_consensus.cc:374] T 00000000000000000000000000000000 P 981fa569ac49410ea40740b68baa95fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:59:19.285457 4951 raft_consensus.cc:729] T 00000000000000000000000000000000 P 981fa569ac49410ea40740b68baa95fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 981fa569ac49410ea40740b68baa95fa, State: Initialized, Role: FOLLOWER
I20260806 02:59:19.286067 4951 consensus_queue.cc:260] T 00000000000000000000000000000000 P 981fa569ac49410ea40740b68baa95fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "981fa569ac49410ea40740b68baa95fa" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 44127 } }
I20260806 02:59:19.286293 4951 raft_consensus.cc:388] T 00000000000000000000000000000000 P 981fa569ac49410ea40740b68baa95fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:59:19.286478 4951 raft_consensus.cc:482] T 00000000000000000000000000000000 P 981fa569ac49410ea40740b68baa95fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:59:19.286759 4951 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 981fa569ac49410ea40740b68baa95fa [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:59:19.289314 4951 raft_consensus.cc:504] T 00000000000000000000000000000000 P 981fa569ac49410ea40740b68baa95fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "981fa569ac49410ea40740b68baa95fa" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 44127 } }
I20260806 02:59:19.290170 4951 leader_election.cc:302] T 00000000000000000000000000000000 P 981fa569ac49410ea40740b68baa95fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 981fa569ac49410ea40740b68baa95fa; no voters:
I20260806 02:59:19.291023 4951 leader_election.cc:288] T 00000000000000000000000000000000 P 981fa569ac49410ea40740b68baa95fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:59:19.291133 4956 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 981fa569ac49410ea40740b68baa95fa [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:59:19.292111 4956 raft_consensus.cc:686] T 00000000000000000000000000000000 P 981fa569ac49410ea40740b68baa95fa [term 1 LEADER]: Becoming Leader. State: Replica: 981fa569ac49410ea40740b68baa95fa, State: Running, Role: LEADER
I20260806 02:59:19.292768 4956 consensus_queue.cc:237] T 00000000000000000000000000000000 P 981fa569ac49410ea40740b68baa95fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "981fa569ac49410ea40740b68baa95fa" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 44127 } }
I20260806 02:59:19.293713 4951 sys_catalog.cc:564] T 00000000000000000000000000000000 P 981fa569ac49410ea40740b68baa95fa [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:59:19.297930 4958 sys_catalog.cc:455] T 00000000000000000000000000000000 P 981fa569ac49410ea40740b68baa95fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 981fa569ac49410ea40740b68baa95fa. Latest consensus state: current_term: 1 leader_uuid: "981fa569ac49410ea40740b68baa95fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "981fa569ac49410ea40740b68baa95fa" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 44127 } } }
I20260806 02:59:19.298630 4958 sys_catalog.cc:458] T 00000000000000000000000000000000 P 981fa569ac49410ea40740b68baa95fa [sys.catalog]: This master's current role is: LEADER
I20260806 02:59:19.298862 4957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 981fa569ac49410ea40740b68baa95fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "981fa569ac49410ea40740b68baa95fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "981fa569ac49410ea40740b68baa95fa" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 44127 } } }
I20260806 02:59:19.299862 4957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 981fa569ac49410ea40740b68baa95fa [sys.catalog]: This master's current role is: LEADER
I20260806 02:59:19.301486 4962 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:59:19.308015 4962 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:59:19.320622 4962 catalog_manager.cc:1347] Generated new cluster ID: efde9e3c5aed4f97a74f664de3af644d
I20260806 02:59:19.320783 4962 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:59:19.339885 4962 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:59:19.341506 4962 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:59:19.351613 4962 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 981fa569ac49410ea40740b68baa95fa: Generated new TSK 0
I20260806 02:59:19.352895 4962 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260806 02:59:19.406415 4983 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260806 02:59:19.406666 4983 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260806 02:59:19.408001 4954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:19.381794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46878 (local address 127.31.74.254:44127)
0806 02:59:19.382025 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:19.382032 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:19.382436 (+ 404us) server_negotiation.cc:407] Connection header received
0806 02:59:19.382876 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:19.382883 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:19.382992 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:19.383208 (+ 216us) server_negotiation.cc:226] Negotiated authn=JWT
0806 02:59:19.383773 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:19.385060 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:19.386985 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:19.387726 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:19.387875 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:19.389165 (+ 1290us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 02:59:19.407253 (+ 18088us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41489/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0806 02:59:19.407671 (+ 418us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46878: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41489/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":93}
W20260806 02:59:19.408149 4954 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46878: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41489/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260806 02:59:19.410027 4982 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:59:19.381593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:44127 (local address 127.0.0.1:46878)
0806 02:59:19.382315 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:19.382338 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:19.382495 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:19.383234 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:19.383239 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:19.383249 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0806 02:59:19.383605 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:19.383613 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:19.385251 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:19.385255 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:19.386827 (+ 1572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:19.386843 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:19.388448 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:19.388462 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:19.388647 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:19.388733 (+ 86us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 02:59:19.407588 (+ 18855us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41489/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0806 02:59:19.407963 (+ 375us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.31.74.254:44127: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41489/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":364,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.31.74.254:44127: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41489/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260806 02:59:19.412611 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 4884
I20260806 02:59:19.414224 4946 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:59:19.529156 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 4884
2026-08-06T02:59:19Z chronyd exiting
I20260806 02:59:19.591915 32043 test_util.cc:175] -----------------------------------------------
I20260806 02:59:19.592077 32043 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785985105255475-32043-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (926 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260806 02:59:19.597188 32043 mini_oidc.cc:152] Starting JWKS server
I20260806 02:59:19.599362 32043 webserver.cc:460] Webserver started at https://127.0.0.1:46289/ using document root <none> and password file <none>
I20260806 02:59:19.599628 32043 mini_oidc.cc:189] Starting OIDC Discovery server
I20260806 02:59:19.600445 32043 webserver.cc:460] Webserver started at http://127.0.0.1:42697/ using document root <none> and password file <none>
2026-08-06T02:59:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:59:19Z Disabled control of system clock
I20260806 02:59:19.655196 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785985105255475-32043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785985105255475-32043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785985105255475-32043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:37417
--webserver_interface=127.31.74.254
--webserver_port=0
--builtin_ntp_servers=127.31.74.212:45195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.74.254:37417
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46289/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785985105255475-32043-0/testchainca.cert with env {}
W20260806 02:59:20.089326 4990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:59:20.089763 4990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:59:20.089937 4990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:59:20.099498 4990 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:59:20.099694 4990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:59:20.099772 4990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:59:20.099871 4990 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:59:20.099963 4990 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 02:59:20.100063 4990 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 02:59:20.100145 4990 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46289/jwks.json
I20260806 02:59:20.112385 4990 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:45195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785985105255475-32043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785985105255475-32043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.74.254:37417
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:37417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46289/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.74.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785985105255475-32043-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785985105255475-32043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:59:20.114042 4990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:59:20.115875 4990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:59:20.134481 4995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:20.134459 4998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:20.134543 4996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:59:20.136660 4990 server_base.cc:1034] running on GCE node
I20260806 02:59:20.138257 4990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:59:20.140017 4990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:59:20.141295 4990 hybrid_clock.cc:648] HybridClock initialized: now 1785985160141259 us; error 38 us; skew 500 ppm
I20260806 02:59:20.141796 4990 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 02:59:20.144924 4990 webserver.cc:460] Webserver started at http://127.31.74.254:37685/ using document root <none> and password file <none>
I20260806 02:59:20.145774 4990 fs_manager.cc:359] Metadata directory not provided
I20260806 02:59:20.145934 4990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:59:20.146320 4990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:59:20.149847 4990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785985105255475-32043-0/minicluster-data/master-0/data/instance:
uuid: "084ac3b9b6764d4fa53f688c2ce617f8"
format_stamp: "Formatted at 2026-08-06 02:59:20 on dist-test-slave-dhph"
server_key: "ad0f7171ebc65e662b1fcfd5a45aa547"
server_key_iv: "1867daac77af1b6237bf7645932381f7"
server_key_version: "encryptionkey@0"
I20260806 02:59:20.151096 4990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785985105255475-32043-0/minicluster-data/master-0/wal/instance:
uuid: "084ac3b9b6764d4fa53f688c2ce617f8"
format_stamp: "Formatted at 2026-08-06 02:59:20 on dist-test-slave-dhph"
server_key: "ad0f7171ebc65e662b1fcfd5a45aa547"
server_key_iv: "1867daac77af1b6237bf7645932381f7"
server_key_version: "encryptionkey@0"
I20260806 02:59:20.157673 4990 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260806 02:59:20.162376 5005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:20.164815 4990 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 02:59:20.165084 4990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785985105255475-32043-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785985105255475-32043-0/minicluster-data/master-0/wal
uuid: "084ac3b9b6764d4fa53f688c2ce617f8"
format_stamp: "Formatted at 2026-08-06 02:59:20 on dist-test-slave-dhph"
server_key: "ad0f7171ebc65e662b1fcfd5a45aa547"
server_key_iv: "1867daac77af1b6237bf7645932381f7"
server_key_version: "encryptionkey@0"
I20260806 02:59:20.165381 4990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785985105255475-32043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785985105255475-32043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785985105255475-32043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:59:20.191226 4990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:59:20.192247 4990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:59:20.192616 4990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:59:20.217145 4990 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.254:37417
I20260806 02:59:20.217167 5056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.254:37417 every 8 connection(s)
I20260806 02:59:20.219412 4990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
I20260806 02:59:20.224593 5057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:59:20.229693 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 4990
I20260806 02:59:20.229924 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785985105255475-32043-0/minicluster-data/master-0/wal/instance
I20260806 02:59:20.230913 32043 external_mini_cluster.cc:1428] Setting key 87255b5bc1ec744c0135e5ff8e708f6d
I20260806 02:59:20.239822 5057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 084ac3b9b6764d4fa53f688c2ce617f8: Bootstrap starting.
I20260806 02:59:20.243969 5057 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 084ac3b9b6764d4fa53f688c2ce617f8: Neither blocks nor log segments found. Creating new log.
I20260806 02:59:20.245183 5057 log.cc:824] T 00000000000000000000000000000000 P 084ac3b9b6764d4fa53f688c2ce617f8: Log is configured to *not* fsync() on all Append() calls
I20260806 02:59:20.248977 5057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 084ac3b9b6764d4fa53f688c2ce617f8: No bootstrap required, opened a new log
I20260806 02:59:20.252403 5060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:20.233723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33538 (local address 127.31.74.254:37417)
0806 02:59:20.235411 (+ 1688us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:20.235424 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:20.235479 (+ 55us) server_negotiation.cc:407] Connection header received
0806 02:59:20.238309 (+ 2830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:20.238341 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:20.239054 (+ 713us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:20.239530 (+ 476us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:20.240166 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:20.242714 (+ 2548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:20.245880 (+ 3166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:20.247008 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:20.247250 (+ 242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:20.248765 (+ 1515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:20.248777 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:20.248799 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:59:20.249685 (+ 886us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:20.249821 (+ 136us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:59:20.249864 (+ 43us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:20.249990 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:20.250883 (+ 893us) server_negotiation.cc:299] Negotiation successful
0806 02:59:20.251253 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1023,"thread_start_us":511,"threads_started":1}
I20260806 02:59:20.260645 32043 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:59:20.260886 5057 raft_consensus.cc:348] T 00000000000000000000000000000000 P 084ac3b9b6764d4fa53f688c2ce617f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "084ac3b9b6764d4fa53f688c2ce617f8" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 37417 } }
I20260806 02:59:20.261343 5057 raft_consensus.cc:374] T 00000000000000000000000000000000 P 084ac3b9b6764d4fa53f688c2ce617f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:59:20.261469 5057 raft_consensus.cc:729] T 00000000000000000000000000000000 P 084ac3b9b6764d4fa53f688c2ce617f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 084ac3b9b6764d4fa53f688c2ce617f8, State: Initialized, Role: FOLLOWER
I20260806 02:59:20.262104 5057 consensus_queue.cc:260] T 00000000000000000000000000000000 P 084ac3b9b6764d4fa53f688c2ce617f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "084ac3b9b6764d4fa53f688c2ce617f8" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 37417 } }
I20260806 02:59:20.262385 5057 raft_consensus.cc:388] T 00000000000000000000000000000000 P 084ac3b9b6764d4fa53f688c2ce617f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:59:20.262552 5057 raft_consensus.cc:482] T 00000000000000000000000000000000 P 084ac3b9b6764d4fa53f688c2ce617f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:59:20.262820 5057 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 084ac3b9b6764d4fa53f688c2ce617f8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:59:20.266541 5057 raft_consensus.cc:504] T 00000000000000000000000000000000 P 084ac3b9b6764d4fa53f688c2ce617f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "084ac3b9b6764d4fa53f688c2ce617f8" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 37417 } }
I20260806 02:59:20.267282 5057 leader_election.cc:302] T 00000000000000000000000000000000 P 084ac3b9b6764d4fa53f688c2ce617f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 084ac3b9b6764d4fa53f688c2ce617f8; no voters:
I20260806 02:59:20.268087 5057 leader_election.cc:288] T 00000000000000000000000000000000 P 084ac3b9b6764d4fa53f688c2ce617f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:59:20.268371 5062 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 084ac3b9b6764d4fa53f688c2ce617f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:59:20.269281 5062 raft_consensus.cc:686] T 00000000000000000000000000000000 P 084ac3b9b6764d4fa53f688c2ce617f8 [term 1 LEADER]: Becoming Leader. State: Replica: 084ac3b9b6764d4fa53f688c2ce617f8, State: Running, Role: LEADER
I20260806 02:59:20.269943 5062 consensus_queue.cc:237] T 00000000000000000000000000000000 P 084ac3b9b6764d4fa53f688c2ce617f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "084ac3b9b6764d4fa53f688c2ce617f8" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 37417 } }
I20260806 02:59:20.270941 5057 sys_catalog.cc:564] T 00000000000000000000000000000000 P 084ac3b9b6764d4fa53f688c2ce617f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:59:20.273262 5063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 084ac3b9b6764d4fa53f688c2ce617f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "084ac3b9b6764d4fa53f688c2ce617f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "084ac3b9b6764d4fa53f688c2ce617f8" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 37417 } } }
I20260806 02:59:20.273667 5063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 084ac3b9b6764d4fa53f688c2ce617f8 [sys.catalog]: This master's current role is: LEADER
I20260806 02:59:20.274816 5067 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:59:20.274626 5064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 084ac3b9b6764d4fa53f688c2ce617f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 084ac3b9b6764d4fa53f688c2ce617f8. Latest consensus state: current_term: 1 leader_uuid: "084ac3b9b6764d4fa53f688c2ce617f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "084ac3b9b6764d4fa53f688c2ce617f8" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 37417 } } }
I20260806 02:59:20.275149 5064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 084ac3b9b6764d4fa53f688c2ce617f8 [sys.catalog]: This master's current role is: LEADER
I20260806 02:59:20.281977 5067 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:59:20.293102 5067 catalog_manager.cc:1347] Generated new cluster ID: fb8873b7b6ac4c9cbd91fe07f9d394c0
I20260806 02:59:20.293272 5067 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:59:20.328756 5067 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:59:20.330691 5067 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:59:20.346818 5067 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 084ac3b9b6764d4fa53f688c2ce617f8: Generated new TSK 0
I20260806 02:59:20.348270 5067 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260806 02:59:20.399027 5088 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260806 02:59:20.399176 5088 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260806 02:59:20.400504 5060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:20.379747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33554 (local address 127.31.74.254:37417)
0806 02:59:20.379969 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:20.379976 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:20.380127 (+ 151us) server_negotiation.cc:407] Connection header received
0806 02:59:20.380382 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:20.380389 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:20.380495 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:20.380668 (+ 173us) server_negotiation.cc:226] Negotiated authn=JWT
0806 02:59:20.381146 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:20.382326 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:20.383997 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:20.384717 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:20.384854 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:20.385377 (+ 523us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 02:59:20.399748 (+ 14371us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46289/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0806 02:59:20.400172 (+ 424us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33554: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46289/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":84}
W20260806 02:59:20.400632 5060 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33554: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46289/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260806 02:59:20.401062 5087 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:59:20.379311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:37417 (local address 127.0.0.1:33554)
0806 02:59:20.380016 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:20.380034 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:20.380172 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:20.380697 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:20.380701 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:20.380711 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0806 02:59:20.380996 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:20.381005 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:20.382508 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:20.382513 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:20.383840 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:20.383855 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:20.384918 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:20.384922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:20.385062 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:20.385100 (+ 38us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 02:59:20.400044 (+ 14944us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46289/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0806 02:59:20.400651 (+ 607us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.31.74.254:37417: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46289/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":372,"threads_started":1}
I20260806 02:59:20.402884 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 4990
I20260806 02:59:20.404139 5052 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:59:20.507565 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 4990
2026-08-06T02:59:20Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (972 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260806 02:59:20.570473 32043 mini_oidc.cc:152] Starting JWKS server
I20260806 02:59:20.573671 32043 webserver.cc:460] Webserver started at https://127.0.0.1:32931/ using document root <none> and password file <none>
I20260806 02:59:20.573915 32043 mini_oidc.cc:189] Starting OIDC Discovery server
I20260806 02:59:20.574608 32043 webserver.cc:460] Webserver started at http://127.0.0.1:46233/ using document root <none> and password file <none>
2026-08-06T02:59:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:59:20Z Disabled control of system clock
I20260806 02:59:20.620894 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785985105255475-32043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785985105255475-32043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785985105255475-32043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:35317
--webserver_interface=127.31.74.254
--webserver_port=0
--builtin_ntp_servers=127.31.74.212:34333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.74.254:35317
--enable_jwt_token_auth=true
--jwks_url=https://localhost:32931/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260806 02:59:20.997102 5096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:59:20.997434 5096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:59:20.997591 5096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:59:21.005081 5096 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:59:21.005410 5096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:59:21.005501 5096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:59:21.005700 5096 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:59:21.005790 5096 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 02:59:21.005957 5096 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 02:59:21.006013 5096 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:32931/jwks.json
I20260806 02:59:21.018359 5096 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:34333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785985105255475-32043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785985105255475-32043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.74.254:35317
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:35317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:32931/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.74.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785985105255475-32043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:59:21.019898 5096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:59:21.021948 5096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:59:21.037158 5104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:21.037062 5102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:21.037398 5101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:59:21.038043 5096 server_base.cc:1034] running on GCE node
I20260806 02:59:21.039027 5096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:59:21.040550 5096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:59:21.041796 5096 hybrid_clock.cc:648] HybridClock initialized: now 1785985161041753 us; error 49 us; skew 500 ppm
I20260806 02:59:21.042168 5096 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 02:59:21.044634 5096 webserver.cc:460] Webserver started at http://127.31.74.254:41263/ using document root <none> and password file <none>
I20260806 02:59:21.045347 5096 fs_manager.cc:359] Metadata directory not provided
I20260806 02:59:21.045485 5096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:59:21.045822 5096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:59:21.048870 5096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785985105255475-32043-0/minicluster-data/master-0/data/instance:
uuid: "ab9f073d01964302a934a0186ae6cfa1"
format_stamp: "Formatted at 2026-08-06 02:59:21 on dist-test-slave-dhph"
server_key: "3123079269a817bf08d909e1ad81d94b"
server_key_iv: "74c96b6539aa09589d9471934ba1c0b7"
server_key_version: "encryptionkey@0"
I20260806 02:59:21.049705 5096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785985105255475-32043-0/minicluster-data/master-0/wal/instance:
uuid: "ab9f073d01964302a934a0186ae6cfa1"
format_stamp: "Formatted at 2026-08-06 02:59:21 on dist-test-slave-dhph"
server_key: "3123079269a817bf08d909e1ad81d94b"
server_key_iv: "74c96b6539aa09589d9471934ba1c0b7"
server_key_version: "encryptionkey@0"
I20260806 02:59:21.055140 5096 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260806 02:59:21.058980 5110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:21.060612 5096 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:59:21.060858 5096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785985105255475-32043-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785985105255475-32043-0/minicluster-data/master-0/wal
uuid: "ab9f073d01964302a934a0186ae6cfa1"
format_stamp: "Formatted at 2026-08-06 02:59:21 on dist-test-slave-dhph"
server_key: "3123079269a817bf08d909e1ad81d94b"
server_key_iv: "74c96b6539aa09589d9471934ba1c0b7"
server_key_version: "encryptionkey@0"
I20260806 02:59:21.061123 5096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785985105255475-32043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785985105255475-32043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785985105255475-32043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:59:21.111992 5096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:59:21.113070 5096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:59:21.113461 5096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:59:21.140970 5096 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.254:35317
I20260806 02:59:21.140986 5162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.254:35317 every 8 connection(s)
I20260806 02:59:21.142664 5096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
I20260806 02:59:21.147297 5163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:59:21.152799 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 5096
I20260806 02:59:21.153059 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785985105255475-32043-0/minicluster-data/master-0/wal/instance
I20260806 02:59:21.153925 32043 external_mini_cluster.cc:1428] Setting key 1b092db843823d9522f323cb87abf361
I20260806 02:59:21.166095 5163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab9f073d01964302a934a0186ae6cfa1: Bootstrap starting.
I20260806 02:59:21.170408 5163 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab9f073d01964302a934a0186ae6cfa1: Neither blocks nor log segments found. Creating new log.
I20260806 02:59:21.170465 5166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:21.156955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57854 (local address 127.31.74.254:35317)
0806 02:59:21.158085 (+ 1130us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:21.158096 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:21.158139 (+ 43us) server_negotiation.cc:407] Connection header received
0806 02:59:21.160451 (+ 2312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:21.160476 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:21.161056 (+ 580us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:21.161520 (+ 464us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:21.162150 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:21.163649 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:21.165031 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:21.165906 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:21.166131 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:21.167724 (+ 1593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:21.167733 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:21.167748 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:59:21.168524 (+ 776us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:21.168619 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:59:21.168643 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:21.168839 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:21.169101 (+ 262us) server_negotiation.cc:299] Negotiation successful
0806 02:59:21.169465 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":665,"thread_start_us":413,"threads_started":1}
I20260806 02:59:21.172796 5163 log.cc:824] T 00000000000000000000000000000000 P ab9f073d01964302a934a0186ae6cfa1: Log is configured to *not* fsync() on all Append() calls
I20260806 02:59:21.176746 32043 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:59:21.177819 5163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab9f073d01964302a934a0186ae6cfa1: No bootstrap required, opened a new log
I20260806 02:59:21.187549 5163 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab9f073d01964302a934a0186ae6cfa1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab9f073d01964302a934a0186ae6cfa1" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 35317 } }
I20260806 02:59:21.188102 5163 raft_consensus.cc:374] T 00000000000000000000000000000000 P ab9f073d01964302a934a0186ae6cfa1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:59:21.188195 5163 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab9f073d01964302a934a0186ae6cfa1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab9f073d01964302a934a0186ae6cfa1, State: Initialized, Role: FOLLOWER
I20260806 02:59:21.188745 5163 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab9f073d01964302a934a0186ae6cfa1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab9f073d01964302a934a0186ae6cfa1" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 35317 } }
I20260806 02:59:21.188980 5163 raft_consensus.cc:388] T 00000000000000000000000000000000 P ab9f073d01964302a934a0186ae6cfa1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:59:21.189082 5163 raft_consensus.cc:482] T 00000000000000000000000000000000 P ab9f073d01964302a934a0186ae6cfa1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:59:21.189218 5163 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab9f073d01964302a934a0186ae6cfa1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:59:21.192029 5163 raft_consensus.cc:504] T 00000000000000000000000000000000 P ab9f073d01964302a934a0186ae6cfa1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab9f073d01964302a934a0186ae6cfa1" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 35317 } }
I20260806 02:59:21.192548 5163 leader_election.cc:302] T 00000000000000000000000000000000 P ab9f073d01964302a934a0186ae6cfa1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ab9f073d01964302a934a0186ae6cfa1; no voters:
I20260806 02:59:21.193161 5163 leader_election.cc:288] T 00000000000000000000000000000000 P ab9f073d01964302a934a0186ae6cfa1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:59:21.193331 5168 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ab9f073d01964302a934a0186ae6cfa1 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:59:21.194080 5168 raft_consensus.cc:686] T 00000000000000000000000000000000 P ab9f073d01964302a934a0186ae6cfa1 [term 1 LEADER]: Becoming Leader. State: Replica: ab9f073d01964302a934a0186ae6cfa1, State: Running, Role: LEADER
I20260806 02:59:21.194722 5168 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab9f073d01964302a934a0186ae6cfa1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab9f073d01964302a934a0186ae6cfa1" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 35317 } }
I20260806 02:59:21.195472 5163 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab9f073d01964302a934a0186ae6cfa1 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:59:21.198155 5170 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab9f073d01964302a934a0186ae6cfa1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab9f073d01964302a934a0186ae6cfa1. Latest consensus state: current_term: 1 leader_uuid: "ab9f073d01964302a934a0186ae6cfa1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab9f073d01964302a934a0186ae6cfa1" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 35317 } } }
I20260806 02:59:21.198707 5170 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab9f073d01964302a934a0186ae6cfa1 [sys.catalog]: This master's current role is: LEADER
I20260806 02:59:21.199779 5169 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab9f073d01964302a934a0186ae6cfa1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ab9f073d01964302a934a0186ae6cfa1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab9f073d01964302a934a0186ae6cfa1" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 35317 } } }
I20260806 02:59:21.200506 5169 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab9f073d01964302a934a0186ae6cfa1 [sys.catalog]: This master's current role is: LEADER
I20260806 02:59:21.200527 5173 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:59:21.206527 5173 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:59:21.216758 5173 catalog_manager.cc:1347] Generated new cluster ID: b55dc5cc33ea48e9ad5636f8bb7205a9
I20260806 02:59:21.216904 5173 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:59:21.239022 5173 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:59:21.241364 5173 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:59:21.249466 5173 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ab9f073d01964302a934a0186ae6cfa1: Generated new TSK 0
I20260806 02:59:21.250463 5173 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:59:21.277518 5166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:21.266770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57858 (local address 127.31.74.254:35317)
0806 02:59:21.266976 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:21.266981 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:21.267585 (+ 604us) server_negotiation.cc:407] Connection header received
0806 02:59:21.267854 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:21.267861 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:21.267972 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:21.268210 (+ 238us) server_negotiation.cc:226] Negotiated authn=JWT
0806 02:59:21.268804 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:21.270229 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:21.272046 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:21.272892 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:21.273059 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:21.273791 (+ 732us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 02:59:21.276792 (+ 3001us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32931/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0806 02:59:21.277198 (+ 406us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57858: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32931/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":49}
W20260806 02:59:21.277669 5166 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57858: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32931/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260806 02:59:21.282202 5193 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:59:21.266721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:35317 (local address 127.0.0.1:57858)
0806 02:59:21.267464 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:21.267482 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:21.267635 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:21.268261 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:21.268265 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:21.268277 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0806 02:59:21.268623 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:21.268633 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:21.270498 (+ 1865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:21.270503 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:21.271844 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:21.271859 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:21.273221 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:21.273227 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:21.273394 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:21.273439 (+ 45us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 02:59:21.277076 (+ 3637us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32931/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0806 02:59:21.280760 (+ 3684us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.31.74.254:35317: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32931/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":585,"mutex_wait_us":2797,"thread_start_us":414,"threads_started":1}
I20260806 02:59:21.283756 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 5096
I20260806 02:59:21.286402 5158 generic_service.cc:196] Checking for leaks (request via RPC)
E20260806 02:59:21.296320 5194 webserver.cc:533] Webserver: sslize failed:
E20260806 02:59:21.296505 5194 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260806 02:59:21.395560 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 5096
2026-08-06T02:59:21Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (887 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-06T02:59:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:59:21Z Disabled control of system clock
I20260806 02:59:21.501628 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785985105255475-32043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785985105255475-32043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785985105255475-32043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:38951
--webserver_interface=127.31.74.254
--webserver_port=0
--builtin_ntp_servers=127.31.74.212:33141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.74.254:38951
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785985105255475-32043-0/testchainca.cert with env {}
W20260806 02:59:21.905957 5199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:59:21.906302 5199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:59:21.906414 5199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:59:21.916638 5199 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:59:21.916806 5199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:59:21.916847 5199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:59:21.916901 5199 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:59:21.916945 5199 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 02:59:21.917003 5199 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 02:59:21.917075 5199 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260806 02:59:21.932209 5199 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:33141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785985105255475-32043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785985105255475-32043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.74.254:38951
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:38951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.74.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785985105255475-32043-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785985105255475-32043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:59:21.933630 5199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:59:21.935065 5199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:59:21.949046 5204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:21.949110 5207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:21.949079 5205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:59:21.950160 5199 server_base.cc:1034] running on GCE node
I20260806 02:59:21.951224 5199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:59:21.952900 5199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:59:21.954178 5199 hybrid_clock.cc:648] HybridClock initialized: now 1785985161954098 us; error 87 us; skew 500 ppm
I20260806 02:59:21.954571 5199 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 02:59:21.957448 5199 webserver.cc:460] Webserver started at http://127.31.74.254:38585/ using document root <none> and password file <none>
I20260806 02:59:21.958179 5199 fs_manager.cc:359] Metadata directory not provided
I20260806 02:59:21.958276 5199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:59:21.958536 5199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:59:21.961647 5199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785985105255475-32043-0/minicluster-data/master-0/data/instance:
uuid: "67b1499f32ea40bcbe14d3293a4f6427"
format_stamp: "Formatted at 2026-08-06 02:59:21 on dist-test-slave-dhph"
server_key: "2db25d19f2170882f03bd26d4ec9bfc9"
server_key_iv: "46fe744e2800258cfb6315fc3bdcb4ca"
server_key_version: "encryptionkey@0"
I20260806 02:59:21.962818 5199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785985105255475-32043-0/minicluster-data/master-0/wal/instance:
uuid: "67b1499f32ea40bcbe14d3293a4f6427"
format_stamp: "Formatted at 2026-08-06 02:59:21 on dist-test-slave-dhph"
server_key: "2db25d19f2170882f03bd26d4ec9bfc9"
server_key_iv: "46fe744e2800258cfb6315fc3bdcb4ca"
server_key_version: "encryptionkey@0"
I20260806 02:59:21.969242 5199 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.003s
I20260806 02:59:21.973544 5213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:21.975519 5199 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260806 02:59:21.975766 5199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785985105255475-32043-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785985105255475-32043-0/minicluster-data/master-0/wal
uuid: "67b1499f32ea40bcbe14d3293a4f6427"
format_stamp: "Formatted at 2026-08-06 02:59:21 on dist-test-slave-dhph"
server_key: "2db25d19f2170882f03bd26d4ec9bfc9"
server_key_iv: "46fe744e2800258cfb6315fc3bdcb4ca"
server_key_version: "encryptionkey@0"
I20260806 02:59:21.976042 5199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785985105255475-32043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785985105255475-32043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785985105255475-32043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:59:21.998332 5199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:59:21.999439 5199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:59:21.999729 5199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:59:22.025238 5265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.254:38951 every 8 connection(s)
I20260806 02:59:22.025281 5199 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.254:38951
I20260806 02:59:22.028456 5199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
I20260806 02:59:22.033316 5266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:59:22.037539 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 5199
I20260806 02:59:22.037775 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785985105255475-32043-0/minicluster-data/master-0/wal/instance
I20260806 02:59:22.038659 32043 external_mini_cluster.cc:1428] Setting key 07987733d83d22a8da11f84764e395e3
I20260806 02:59:22.045678 5266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67b1499f32ea40bcbe14d3293a4f6427: Bootstrap starting.
I20260806 02:59:22.049839 5266 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67b1499f32ea40bcbe14d3293a4f6427: Neither blocks nor log segments found. Creating new log.
I20260806 02:59:22.051437 5266 log.cc:824] T 00000000000000000000000000000000 P 67b1499f32ea40bcbe14d3293a4f6427: Log is configured to *not* fsync() on all Append() calls
I20260806 02:59:22.055550 5266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67b1499f32ea40bcbe14d3293a4f6427: No bootstrap required, opened a new log
I20260806 02:59:22.055958 5269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:22.040755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48598 (local address 127.31.74.254:38951)
0806 02:59:22.041733 (+ 978us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:22.041744 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:22.041801 (+ 57us) server_negotiation.cc:407] Connection header received
0806 02:59:22.044399 (+ 2598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:22.044425 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:22.044932 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:22.045301 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:22.046005 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:22.047795 (+ 1790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:22.050327 (+ 2532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:22.051152 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:22.051321 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:22.052880 (+ 1559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:22.052896 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:22.052923 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:59:22.053742 (+ 819us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:22.053833 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:59:22.053857 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:22.053954 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:22.054327 (+ 373us) server_negotiation.cc:299] Negotiation successful
0806 02:59:22.054802 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":379,"threads_started":1}
I20260806 02:59:22.061560 32043 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:59:22.068421 5266 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67b1499f32ea40bcbe14d3293a4f6427 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67b1499f32ea40bcbe14d3293a4f6427" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 38951 } }
I20260806 02:59:22.068841 5266 raft_consensus.cc:374] T 00000000000000000000000000000000 P 67b1499f32ea40bcbe14d3293a4f6427 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:59:22.068924 5266 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67b1499f32ea40bcbe14d3293a4f6427 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67b1499f32ea40bcbe14d3293a4f6427, State: Initialized, Role: FOLLOWER
I20260806 02:59:22.069545 5266 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67b1499f32ea40bcbe14d3293a4f6427 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67b1499f32ea40bcbe14d3293a4f6427" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 38951 } }
I20260806 02:59:22.069761 5266 raft_consensus.cc:388] T 00000000000000000000000000000000 P 67b1499f32ea40bcbe14d3293a4f6427 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:59:22.069868 5266 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67b1499f32ea40bcbe14d3293a4f6427 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:59:22.070015 5266 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67b1499f32ea40bcbe14d3293a4f6427 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:59:22.073145 5266 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67b1499f32ea40bcbe14d3293a4f6427 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67b1499f32ea40bcbe14d3293a4f6427" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 38951 } }
I20260806 02:59:22.073837 5266 leader_election.cc:302] T 00000000000000000000000000000000 P 67b1499f32ea40bcbe14d3293a4f6427 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 67b1499f32ea40bcbe14d3293a4f6427; no voters:
I20260806 02:59:22.074517 5266 leader_election.cc:288] T 00000000000000000000000000000000 P 67b1499f32ea40bcbe14d3293a4f6427 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:59:22.074756 5271 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67b1499f32ea40bcbe14d3293a4f6427 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:59:22.075727 5271 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67b1499f32ea40bcbe14d3293a4f6427 [term 1 LEADER]: Becoming Leader. State: Replica: 67b1499f32ea40bcbe14d3293a4f6427, State: Running, Role: LEADER
I20260806 02:59:22.076593 5271 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67b1499f32ea40bcbe14d3293a4f6427 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67b1499f32ea40bcbe14d3293a4f6427" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 38951 } }
I20260806 02:59:22.077306 5266 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67b1499f32ea40bcbe14d3293a4f6427 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:59:22.080204 5273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67b1499f32ea40bcbe14d3293a4f6427 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67b1499f32ea40bcbe14d3293a4f6427. Latest consensus state: current_term: 1 leader_uuid: "67b1499f32ea40bcbe14d3293a4f6427" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67b1499f32ea40bcbe14d3293a4f6427" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 38951 } } }
I20260806 02:59:22.080643 5273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67b1499f32ea40bcbe14d3293a4f6427 [sys.catalog]: This master's current role is: LEADER
I20260806 02:59:22.081312 5272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67b1499f32ea40bcbe14d3293a4f6427 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "67b1499f32ea40bcbe14d3293a4f6427" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67b1499f32ea40bcbe14d3293a4f6427" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 38951 } } }
I20260806 02:59:22.081760 5272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67b1499f32ea40bcbe14d3293a4f6427 [sys.catalog]: This master's current role is: LEADER
I20260806 02:59:22.087862 5278 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:59:22.093590 5278 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:59:22.103940 5278 catalog_manager.cc:1347] Generated new cluster ID: a6f601c441db42d6b3a6a49afcc3ac88
I20260806 02:59:22.104099 5278 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:59:22.120217 5278 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:59:22.122745 5278 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:59:22.136168 5278 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 67b1499f32ea40bcbe14d3293a4f6427: Generated new TSK 0
I20260806 02:59:22.137432 5278 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:59:22.228215 5269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:22.158246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48608 (local address 127.31.74.254:38951)
0806 02:59:22.158487 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:22.158494 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:22.159422 (+ 928us) server_negotiation.cc:407] Connection header received
0806 02:59:22.159697 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:22.159704 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:22.159844 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:22.160328 (+ 484us) server_negotiation.cc:226] Negotiated authn=JWT
0806 02:59:22.161229 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:22.163236 (+ 2007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:22.164922 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:22.165655 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:22.165823 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:22.166821 (+ 998us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 02:59:22.227326 (+ 60505us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0806 02:59:22.227837 (+ 511us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48608: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":80}
W20260806 02:59:22.228389 5269 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48608: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260806 02:59:22.228482 5296 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:59:22.158160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:38951 (local address 127.0.0.1:48608)
0806 02:59:22.159251 (+ 1091us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:22.159269 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:22.159469 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:22.160309 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:22.160313 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:22.160325 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0806 02:59:22.160628 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:22.160637 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:22.163528 (+ 2891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:22.163534 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:22.164718 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:22.164733 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:22.166145 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:22.166157 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:22.166372 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:22.166429 (+ 57us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 02:59:22.227662 (+ 61233us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0806 02:59:22.228001 (+ 339us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.31.74.254:38951: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":936,"thread_start_us":522,"threads_started":1}
I20260806 02:59:22.230741 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 5199
I20260806 02:59:22.232005 5261 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:59:22.356164 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 5199
2026-08-06T02:59:22Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (939 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (471 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (466 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:59:23 dist-test-slave-dhph krb5kdc[5312](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:59:23 dist-test-slave-dhph krb5kdc[5312](info): set up 2 sockets
Aug 06 02:59:23 dist-test-slave-dhph krb5kdc[5312](info): commencing operation
krb5kdc: starting...
W20260806 02:59:24.528461 32043 mini_kdc.cc:121] Time spent starting KDC: real 1.193s user 0.002s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/krb5kdc/test-user.keytab.
Aug 06 02:59:24 dist-test-slave-dhph krb5kdc[5312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985164, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:59:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:59:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.74.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab.
Entry for principal kudu/127.31.74.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab.
I20260806 02:59:24.852413 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:40053
--webserver_interface=127.31.74.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab
--principal=kudu/127.31.74.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:38289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.74.254:40053
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:59:25.260257 5325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:59:25.260602 5325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:59:25.260785 5325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:59:25.270224 5325 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:59:25.270382 5325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:59:25.270437 5325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:59:25.270634 5325 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:59:25.270707 5325 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:59:25.282052 5325 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:38289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.74.254:40053
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab
--principal=kudu/127.31.74.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:40053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.74.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:59:25.283463 5325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:59:25.284735 5325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:59:25.297549 5330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:25.297585 5333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:25.297720 5331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:59:25.299024 5325 server_base.cc:1034] running on GCE node
I20260806 02:59:25.299860 5325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:59:25.301635 5325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:59:25.302937 5325 hybrid_clock.cc:648] HybridClock initialized: now 1785985165302948 us; error 106 us; skew 500 ppm
I20260806 02:59:25.303397 5325 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:59:25 dist-test-slave-dhph krb5kdc[5312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985165, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:59:25.307072 5325 init.cc:377] Logged in from keytab as kudu/127.31.74.254@KRBTEST.COM (short username kudu)
I20260806 02:59:25.309716 5325 webserver.cc:460] Webserver started at http://127.31.74.254:46263/ using document root <none> and password file <none>
I20260806 02:59:25.310902 5325 fs_manager.cc:359] Metadata directory not provided
I20260806 02:59:25.311101 5325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:59:25.311477 5325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:59:25.315374 5325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/minicluster-data/master-0/data/instance:
uuid: "71f642abff084cfebada4869e5aad999"
format_stamp: "Formatted at 2026-08-06 02:59:25 on dist-test-slave-dhph"
server_key: "d02deeada607f2e5c123ce40a0efac89"
server_key_iv: "53099ceb9bf3f6dec22b453c58d90d4b"
server_key_version: "encryptionkey@0"
I20260806 02:59:25.316341 5325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/minicluster-data/master-0/wal/instance:
uuid: "71f642abff084cfebada4869e5aad999"
format_stamp: "Formatted at 2026-08-06 02:59:25 on dist-test-slave-dhph"
server_key: "d02deeada607f2e5c123ce40a0efac89"
server_key_iv: "53099ceb9bf3f6dec22b453c58d90d4b"
server_key_version: "encryptionkey@0"
I20260806 02:59:25.322412 5325 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260806 02:59:25.326768 5340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:25.328334 5325 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:59:25.328635 5325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/minicluster-data/master-0/wal
uuid: "71f642abff084cfebada4869e5aad999"
format_stamp: "Formatted at 2026-08-06 02:59:25 on dist-test-slave-dhph"
server_key: "d02deeada607f2e5c123ce40a0efac89"
server_key_iv: "53099ceb9bf3f6dec22b453c58d90d4b"
server_key_version: "encryptionkey@0"
I20260806 02:59:25.328877 5325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:59:25.360816 5325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:59:25.366719 5325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:59:25.367103 5325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:59:25.393142 5325 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.254:40053
I20260806 02:59:25.393064 5392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.254:40053 every 8 connection(s)
I20260806 02:59:25.394737 5325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
I20260806 02:59:25.396193 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 5325
I20260806 02:59:25.396427 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/minicluster-data/master-0/wal/instance
I20260806 02:59:25.397284 32043 external_mini_cluster.cc:1428] Setting key fa07c4878c2dd8cfeb09e46a8ac586a3
I20260806 02:59:25.400349 5393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 06 02:59:25 dist-test-slave-dhph krb5kdc[5312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985164, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 02:59:25.414727 5393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71f642abff084cfebada4869e5aad999: Bootstrap starting.
I20260806 02:59:25.419492 5393 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71f642abff084cfebada4869e5aad999: Neither blocks nor log segments found. Creating new log.
I20260806 02:59:25.421164 5393 log.cc:824] T 00000000000000000000000000000000 P 71f642abff084cfebada4869e5aad999: Log is configured to *not* fsync() on all Append() calls
I20260806 02:59:25.424922 5393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71f642abff084cfebada4869e5aad999: No bootstrap required, opened a new log
I20260806 02:59:25.425256 5396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:25.399573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49440 (local address 127.31.74.254:40053)
0806 02:59:25.401003 (+ 1430us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:25.401024 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:25.401101 (+ 77us) server_negotiation.cc:407] Connection header received
0806 02:59:25.403460 (+ 2359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:25.403481 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:25.403919 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:25.404351 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:25.406197 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:25.408102 (+ 1905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:25.409572 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:25.410378 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:25.410648 (+ 270us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:25.415753 (+ 5105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:25.415766 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:25.415786 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:25.415850 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:25.420575 (+ 4725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:25.421556 (+ 981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:25.421563 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:25.421570 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:25.421758 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:25.422201 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:25.422213 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:25.422215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:25.422957 (+ 742us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:25.423281 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:25.423748 (+ 467us) server_negotiation.cc:299] Negotiation successful
0806 02:59:25.424201 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1045,"thread_start_us":494,"threads_started":1}
I20260806 02:59:25.435905 5393 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71f642abff084cfebada4869e5aad999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71f642abff084cfebada4869e5aad999" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 40053 } }
I20260806 02:59:25.436517 5393 raft_consensus.cc:374] T 00000000000000000000000000000000 P 71f642abff084cfebada4869e5aad999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:59:25.436638 5393 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71f642abff084cfebada4869e5aad999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71f642abff084cfebada4869e5aad999, State: Initialized, Role: FOLLOWER
I20260806 02:59:25.437294 5393 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71f642abff084cfebada4869e5aad999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71f642abff084cfebada4869e5aad999" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 40053 } }
I20260806 02:59:25.437639 5393 raft_consensus.cc:388] T 00000000000000000000000000000000 P 71f642abff084cfebada4869e5aad999 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:59:25.437819 5393 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71f642abff084cfebada4869e5aad999 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:59:25.438056 5393 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71f642abff084cfebada4869e5aad999 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:59:25.440683 5393 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71f642abff084cfebada4869e5aad999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71f642abff084cfebada4869e5aad999" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 40053 } }
I20260806 02:59:25.441272 5393 leader_election.cc:302] T 00000000000000000000000000000000 P 71f642abff084cfebada4869e5aad999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 71f642abff084cfebada4869e5aad999; no voters:
I20260806 02:59:25.441910 5393 leader_election.cc:288] T 00000000000000000000000000000000 P 71f642abff084cfebada4869e5aad999 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:59:25.442062 5398 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71f642abff084cfebada4869e5aad999 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:59:25.443389 5398 raft_consensus.cc:686] T 00000000000000000000000000000000 P 71f642abff084cfebada4869e5aad999 [term 1 LEADER]: Becoming Leader. State: Replica: 71f642abff084cfebada4869e5aad999, State: Running, Role: LEADER
I20260806 02:59:25.443950 5398 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71f642abff084cfebada4869e5aad999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71f642abff084cfebada4869e5aad999" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 40053 } }
I20260806 02:59:25.444416 5393 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71f642abff084cfebada4869e5aad999 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:59:25.447016 5399 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71f642abff084cfebada4869e5aad999 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "71f642abff084cfebada4869e5aad999" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71f642abff084cfebada4869e5aad999" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 40053 } } }
I20260806 02:59:25.448210 5399 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71f642abff084cfebada4869e5aad999 [sys.catalog]: This master's current role is: LEADER
I20260806 02:59:25.447950 5400 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71f642abff084cfebada4869e5aad999 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71f642abff084cfebada4869e5aad999. Latest consensus state: current_term: 1 leader_uuid: "71f642abff084cfebada4869e5aad999" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71f642abff084cfebada4869e5aad999" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 40053 } } }
I20260806 02:59:25.448846 5400 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71f642abff084cfebada4869e5aad999 [sys.catalog]: This master's current role is: LEADER
I20260806 02:59:25.455300 5408 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:59:25.460852 5408 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:59:25.473264 5408 catalog_manager.cc:1347] Generated new cluster ID: 8e6b29fb41a04421af229cf9cade3fca
I20260806 02:59:25.473431 5408 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:59:25.491163 5408 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:59:25.492681 5408 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:59:25.501328 5408 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 71f642abff084cfebada4869e5aad999: Generated new TSK 0
I20260806 02:59:25.502501 5408 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.74.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.193.keytab.
Entry for principal kudu/127.31.74.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.193.keytab.
I20260806 02:59:25.576610 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.193:0
--local_ip_for_outbound_sockets=127.31.74.193
--webserver_interface=127.31.74.193
--webserver_port=0
--tserver_master_addrs=127.31.74.254:40053
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.193.keytab
--principal=kudu/127.31.74.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:38289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:59:25.993875 5420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:59:25.994253 5420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:59:25.994443 5420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:59:26.003012 5420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:59:26.003141 5420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:59:26.003387 5420 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.193
I20260806 02:59:26.013998 5420 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:38289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.193.keytab
--principal=kudu/127.31.74.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.74.193
--webserver_port=0
--tserver_master_addrs=127.31.74.254:40053
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.193
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:59:26.015463 5420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:59:26.016839 5420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:59:26.030699 5425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:26.031111 5428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:26.031611 5426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:59:26.032653 5420 server_base.cc:1034] running on GCE node
I20260806 02:59:26.033650 5420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:59:26.034982 5420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:59:26.036314 5420 hybrid_clock.cc:648] HybridClock initialized: now 1785985166036267 us; error 58 us; skew 500 ppm
I20260806 02:59:26.036717 5420 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:59:26 dist-test-slave-dhph krb5kdc[5312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985166, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:59:26.040710 5420 init.cc:377] Logged in from keytab as kudu/127.31.74.193@KRBTEST.COM (short username kudu)
I20260806 02:59:26.043051 5420 webserver.cc:460] Webserver started at http://127.31.74.193:35047/ using document root <none> and password file <none>
I20260806 02:59:26.043848 5420 fs_manager.cc:359] Metadata directory not provided
I20260806 02:59:26.044009 5420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:59:26.044587 5420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:59:26.048195 5420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/minicluster-data/ts-0/data/instance:
uuid: "9372b6ccbeb54da1a42c41324e14aa16"
format_stamp: "Formatted at 2026-08-06 02:59:26 on dist-test-slave-dhph"
server_key: "dca211de90079a853dceee0a13d213d4"
server_key_iv: "223d861e664b5089e136d9863b8168e4"
server_key_version: "encryptionkey@0"
I20260806 02:59:26.049358 5420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/minicluster-data/ts-0/wal/instance:
uuid: "9372b6ccbeb54da1a42c41324e14aa16"
format_stamp: "Formatted at 2026-08-06 02:59:26 on dist-test-slave-dhph"
server_key: "dca211de90079a853dceee0a13d213d4"
server_key_iv: "223d861e664b5089e136d9863b8168e4"
server_key_version: "encryptionkey@0"
I20260806 02:59:26.056383 5420 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20260806 02:59:26.060789 5436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:26.062994 5420 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260806 02:59:26.063244 5420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/minicluster-data/ts-0/wal
uuid: "9372b6ccbeb54da1a42c41324e14aa16"
format_stamp: "Formatted at 2026-08-06 02:59:26 on dist-test-slave-dhph"
server_key: "dca211de90079a853dceee0a13d213d4"
server_key_iv: "223d861e664b5089e136d9863b8168e4"
server_key_version: "encryptionkey@0"
I20260806 02:59:26.063501 5420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:59:26.085562 5420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:59:26.091089 5420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:59:26.091450 5420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:59:26.092574 5420 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:59:26.094313 5420 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:59:26.094444 5420 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:26.094671 5420 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:59:26.094762 5420 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:26.139905 5420 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.193:40647
I20260806 02:59:26.139946 5549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.193:40647 every 8 connection(s)
I20260806 02:59:26.141819 5420 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
I20260806 02:59:26.151928 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 5420
I20260806 02:59:26.152220 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/minicluster-data/ts-0/wal/instance
I20260806 02:59:26.152967 32043 external_mini_cluster.cc:1428] Setting key f6883bf4ba2db0af17e4c42039f839fe
Aug 06 02:59:26 dist-test-slave-dhph krb5kdc[5312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985166, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.193@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 02:59:26.168790 5552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:26.145333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.193:56137 (local address 127.31.74.254:40053)
0806 02:59:26.145923 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:26.145932 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:26.146875 (+ 943us) server_negotiation.cc:407] Connection header received
0806 02:59:26.149024 (+ 2149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:26.149031 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:26.149139 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:26.149277 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:26.151190 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:26.152019 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:26.154369 (+ 2350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:26.154977 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:26.155140 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:26.160405 (+ 5265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:26.160415 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:26.160423 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:26.160479 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:26.164900 (+ 4421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:26.165903 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:26.165911 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:26.165917 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:26.166089 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:26.166744 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:26.166755 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:26.166757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:26.167081 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:26.167279 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:26.168077 (+ 798us) server_negotiation.cc:299] Negotiation successful
0806 02:59:26.168355 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":311,"threads_started":1}
I20260806 02:59:26.169920 5553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:26.145716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:40053 (local address 127.31.74.193:56137)
0806 02:59:26.146536 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:26.146656 (+ 120us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:26.148698 (+ 2042us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:26.149564 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:26.149573 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:26.150183 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:59:26.150983 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:26.151004 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:26.152221 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:26.152226 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:26.154198 (+ 1972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:26.154212 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:26.155413 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:26.155422 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:26.155785 (+ 363us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:26.156659 (+ 874us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:59:26.156699 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:59:26.160107 (+ 3408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:59:26.165140 (+ 5033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:26.165150 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:26.165167 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:26.165741 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:26.166262 (+ 521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:26.166266 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:26.166269 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:26.166511 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:26.167236 (+ 725us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:59:26.167247 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:59:26.167663 (+ 416us) client_negotiation.cc:769] Sending connection context
0806 02:59:26.167972 (+ 309us) client_negotiation.cc:240] Negotiation successful
0806 02:59:26.168581 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":307,"threads_started":1}
I20260806 02:59:26.171976 5550 heartbeater.cc:347] Connected to a master server at 127.31.74.254:40053
I20260806 02:59:26.172356 5550 heartbeater.cc:464] Registering TS with master...
I20260806 02:59:26.173460 5550 heartbeater.cc:511] Master 127.31.74.254:40053 requested a full tablet report, sending...
I20260806 02:59:26.176373 5358 ts_manager.cc:194] Registered new tserver with Master: 9372b6ccbeb54da1a42c41324e14aa16 (127.31.74.193:40647)
I20260806 02:59:26.178897 5358 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.74.193@KRBTEST.COM'} at 127.31.74.193:56137
WARNING: no policy specified for kudu/127.31.74.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.194.keytab.
Entry for principal kudu/127.31.74.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.194.keytab.
I20260806 02:59:26.233294 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.194:0
--local_ip_for_outbound_sockets=127.31.74.194
--webserver_interface=127.31.74.194
--webserver_port=0
--tserver_master_addrs=127.31.74.254:40053
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.194.keytab
--principal=kudu/127.31.74.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:38289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:59:26.636272 5556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:59:26.636580 5556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:59:26.636679 5556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:59:26.645191 5556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:59:26.645399 5556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:59:26.645586 5556 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.194
I20260806 02:59:26.657747 5556 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:38289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.194.keytab
--principal=kudu/127.31.74.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.74.194
--webserver_port=0
--tserver_master_addrs=127.31.74.254:40053
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.194
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:59:26.659642 5556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:59:26.661347 5556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:59:26.678768 5562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:26.678807 5564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:26.678773 5561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:59:26.679514 5556 server_base.cc:1034] running on GCE node
I20260806 02:59:26.680366 5556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:59:26.681420 5556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:59:26.682690 5556 hybrid_clock.cc:648] HybridClock initialized: now 1785985166682611 us; error 80 us; skew 500 ppm
I20260806 02:59:26.683010 5556 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:59:26 dist-test-slave-dhph krb5kdc[5312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985166, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:59:26.686539 5556 init.cc:377] Logged in from keytab as kudu/127.31.74.194@KRBTEST.COM (short username kudu)
I20260806 02:59:26.688235 5556 webserver.cc:460] Webserver started at http://127.31.74.194:34841/ using document root <none> and password file <none>
I20260806 02:59:26.688975 5556 fs_manager.cc:359] Metadata directory not provided
I20260806 02:59:26.689095 5556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:59:26.689391 5556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:59:26.692715 5556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/minicluster-data/ts-1/data/instance:
uuid: "f952b6bed112428f9d14bf8f649bb124"
format_stamp: "Formatted at 2026-08-06 02:59:26 on dist-test-slave-dhph"
server_key: "c7b6096dd429f1e3c00ed7265ad9205e"
server_key_iv: "c53346589c901a62d592184d1d3b9600"
server_key_version: "encryptionkey@0"
I20260806 02:59:26.693562 5556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/minicluster-data/ts-1/wal/instance:
uuid: "f952b6bed112428f9d14bf8f649bb124"
format_stamp: "Formatted at 2026-08-06 02:59:26 on dist-test-slave-dhph"
server_key: "c7b6096dd429f1e3c00ed7265ad9205e"
server_key_iv: "c53346589c901a62d592184d1d3b9600"
server_key_version: "encryptionkey@0"
I20260806 02:59:26.699602 5556 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260806 02:59:26.703970 5572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:26.705554 5556 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260806 02:59:26.705799 5556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/minicluster-data/ts-1/wal
uuid: "f952b6bed112428f9d14bf8f649bb124"
format_stamp: "Formatted at 2026-08-06 02:59:26 on dist-test-slave-dhph"
server_key: "c7b6096dd429f1e3c00ed7265ad9205e"
server_key_iv: "c53346589c901a62d592184d1d3b9600"
server_key_version: "encryptionkey@0"
I20260806 02:59:26.706053 5556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:59:26.725174 5556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:59:26.731403 5556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:59:26.731701 5556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:59:26.732961 5556 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:59:26.734710 5556 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:59:26.734797 5556 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:26.734900 5556 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:59:26.734977 5556 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:26.783397 5556 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.194:37359
I20260806 02:59:26.783435 5688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.194:37359 every 8 connection(s)
I20260806 02:59:26.785051 5556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
I20260806 02:59:26.789315 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 5556
I20260806 02:59:26.789543 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/minicluster-data/ts-1/wal/instance
I20260806 02:59:26.790294 32043 external_mini_cluster.cc:1428] Setting key ed9c2347fe03dbc9ea24fd0c70f30a74
Aug 06 02:59:26 dist-test-slave-dhph krb5kdc[5312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985166, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.194@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 02:59:26.817133 5691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:26.788120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.194:41835 (local address 127.31.74.254:40053)
0806 02:59:26.788608 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:26.788615 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:26.789929 (+ 1314us) server_negotiation.cc:407] Connection header received
0806 02:59:26.792765 (+ 2836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:26.792775 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:26.792892 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:26.793007 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:26.796178 (+ 3171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:26.796902 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:26.799923 (+ 3021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:26.800387 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:26.800516 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:26.805852 (+ 5336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:26.805863 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:26.805871 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:26.805931 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:26.811203 (+ 5272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:26.812845 (+ 1642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:26.812854 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:26.812860 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:26.813048 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:26.814485 (+ 1437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:26.814494 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:26.814500 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:26.815063 (+ 563us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:26.815251 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:26.816286 (+ 1035us) server_negotiation.cc:299] Negotiation successful
0806 02:59:26.816612 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":228,"threads_started":1}
I20260806 02:59:26.817834 5692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:26.788564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:40053 (local address 127.31.74.194:41835)
0806 02:59:26.789700 (+ 1136us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:26.789765 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:26.792395 (+ 2630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:26.793457 (+ 1062us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:26.793471 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:26.794529 (+ 1058us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:59:26.795907 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:26.795942 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:26.797087 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:26.797095 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:26.799675 (+ 2580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:26.799700 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:26.800670 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:26.800681 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:26.801061 (+ 380us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:26.801980 (+ 919us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:59:26.802019 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:59:26.805439 (+ 3420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:59:26.811507 (+ 6068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:26.811517 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:26.811532 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:26.812634 (+ 1102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:26.813673 (+ 1039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:26.813677 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:26.813681 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:26.813949 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:26.815275 (+ 1326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:59:26.815283 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:59:26.815817 (+ 534us) client_negotiation.cc:769] Sending connection context
0806 02:59:26.816090 (+ 273us) client_negotiation.cc:240] Negotiation successful
0806 02:59:26.816566 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":768,"thread_start_us":458,"threads_started":1}
I20260806 02:59:26.819671 5689 heartbeater.cc:347] Connected to a master server at 127.31.74.254:40053
I20260806 02:59:26.820271 5689 heartbeater.cc:464] Registering TS with master...
I20260806 02:59:26.821357 5689 heartbeater.cc:511] Master 127.31.74.254:40053 requested a full tablet report, sending...
I20260806 02:59:26.823405 5358 ts_manager.cc:194] Registered new tserver with Master: f952b6bed112428f9d14bf8f649bb124 (127.31.74.194:37359)
I20260806 02:59:26.824358 5358 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.74.194@KRBTEST.COM'} at 127.31.74.194:41835
WARNING: no policy specified for kudu/127.31.74.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.195.keytab.
Entry for principal kudu/127.31.74.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.195.keytab.
I20260806 02:59:26.872951 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.195:0
--local_ip_for_outbound_sockets=127.31.74.195
--webserver_interface=127.31.74.195
--webserver_port=0
--tserver_master_addrs=127.31.74.254:40053
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.195.keytab
--principal=kudu/127.31.74.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:38289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:59:27.183738 5550 heartbeater.cc:503] Master 127.31.74.254:40053 was elected leader, sending a full tablet report...
W20260806 02:59:27.289206 5695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:59:27.289585 5695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:59:27.289700 5695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:59:27.300088 5695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:59:27.300239 5695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:59:27.300384 5695 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.195
I20260806 02:59:27.314703 5695 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:38289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.195.keytab
--principal=kudu/127.31.74.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.74.195
--webserver_port=0
--tserver_master_addrs=127.31.74.254:40053
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.195
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:59:27.316361 5695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:59:27.318040 5695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:59:27.335764 5700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:27.336068 5702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:27.336622 5704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:59:27.339298 5695 server_base.cc:1034] running on GCE node
I20260806 02:59:27.340124 5695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:59:27.341773 5695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:59:27.343067 5695 hybrid_clock.cc:648] HybridClock initialized: now 1785985167342989 us; error 74 us; skew 500 ppm
I20260806 02:59:27.343431 5695 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:59:27 dist-test-slave-dhph krb5kdc[5312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985167, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:59:27.352617 5695 init.cc:377] Logged in from keytab as kudu/127.31.74.195@KRBTEST.COM (short username kudu)
I20260806 02:59:27.355959 5695 webserver.cc:460] Webserver started at http://127.31.74.195:46779/ using document root <none> and password file <none>
I20260806 02:59:27.356894 5695 fs_manager.cc:359] Metadata directory not provided
I20260806 02:59:27.357033 5695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:59:27.357411 5695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:59:27.361312 5695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/minicluster-data/ts-2/data/instance:
uuid: "510ad74fb7a042edbad175d7dd0db80f"
format_stamp: "Formatted at 2026-08-06 02:59:27 on dist-test-slave-dhph"
server_key: "b2ef408b03f9bc1da50dc7112d894a68"
server_key_iv: "db0a6de6c54296bf9e8588d1f2e045b4"
server_key_version: "encryptionkey@0"
I20260806 02:59:27.362304 5695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/minicluster-data/ts-2/wal/instance:
uuid: "510ad74fb7a042edbad175d7dd0db80f"
format_stamp: "Formatted at 2026-08-06 02:59:27 on dist-test-slave-dhph"
server_key: "b2ef408b03f9bc1da50dc7112d894a68"
server_key_iv: "db0a6de6c54296bf9e8588d1f2e045b4"
server_key_version: "encryptionkey@0"
I20260806 02:59:27.369405 5695 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260806 02:59:27.375017 5711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:27.377388 5695 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260806 02:59:27.377656 5695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/minicluster-data/ts-2/wal
uuid: "510ad74fb7a042edbad175d7dd0db80f"
format_stamp: "Formatted at 2026-08-06 02:59:27 on dist-test-slave-dhph"
server_key: "b2ef408b03f9bc1da50dc7112d894a68"
server_key_iv: "db0a6de6c54296bf9e8588d1f2e045b4"
server_key_version: "encryptionkey@0"
I20260806 02:59:27.377933 5695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:59:27.401110 5695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:59:27.407585 5695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:59:27.408159 5695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:59:27.409513 5695 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:59:27.411971 5695 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:59:27.412118 5695 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:27.412297 5695 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:59:27.412392 5695 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:27.468888 5695 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.195:42721
I20260806 02:59:27.468930 5825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.195:42721 every 8 connection(s)
I20260806 02:59:27.470726 5695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
I20260806 02:59:27.478446 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 5695
I20260806 02:59:27.478747 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/minicluster-data/ts-2/wal/instance
I20260806 02:59:27.479506 32043 external_mini_cluster.cc:1428] Setting key 98c56aa129d396378f27ed3b07a36042
Aug 06 02:59:27 dist-test-slave-dhph krb5kdc[5312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985167, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.195@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 02:59:27.498806 5829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:27.475237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.195:44013 (local address 127.31.74.254:40053)
0806 02:59:27.475800 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:27.475806 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:27.477333 (+ 1527us) server_negotiation.cc:407] Connection header received
0806 02:59:27.479935 (+ 2602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:27.479944 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:27.480049 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:27.480188 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:27.482175 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:27.483125 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:27.485307 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:27.485846 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:27.486002 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:27.491534 (+ 5532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:27.491541 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:27.491544 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:27.491586 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:27.494695 (+ 3109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:27.495857 (+ 1162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:27.495861 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:27.495864 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:27.496101 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:27.496736 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:27.496743 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:27.496747 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:27.497079 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:27.497268 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:27.498053 (+ 785us) server_negotiation.cc:299] Negotiation successful
0806 02:59:27.498309 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":292,"threads_started":1}
I20260806 02:59:27.499877 5830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:27.475716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:40053 (local address 127.31.74.195:44013)
0806 02:59:27.477110 (+ 1394us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:27.477167 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:27.479475 (+ 2308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:27.480501 (+ 1026us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:27.480512 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:27.481238 (+ 726us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:59:27.481964 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:27.481992 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:27.483322 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:27.483327 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:27.485055 (+ 1728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:27.485071 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:27.486119 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:27.486127 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:27.486431 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:27.487593 (+ 1162us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:59:27.487631 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:59:27.491214 (+ 3583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:59:27.494911 (+ 3697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:27.494921 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:27.494936 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:27.495635 (+ 699us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:27.496252 (+ 617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:27.496259 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:27.496265 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:27.496615 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:27.497237 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:59:27.497247 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:59:27.497656 (+ 409us) client_negotiation.cc:769] Sending connection context
0806 02:59:27.497966 (+ 310us) client_negotiation.cc:240] Negotiation successful
0806 02:59:27.498617 (+ 651us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1091,"thread_start_us":914,"threads_started":1}
I20260806 02:59:27.501427 5826 heartbeater.cc:347] Connected to a master server at 127.31.74.254:40053
I20260806 02:59:27.501806 5826 heartbeater.cc:464] Registering TS with master...
I20260806 02:59:27.503170 5826 heartbeater.cc:511] Master 127.31.74.254:40053 requested a full tablet report, sending...
I20260806 02:59:27.505039 5358 ts_manager.cc:194] Registered new tserver with Master: 510ad74fb7a042edbad175d7dd0db80f (127.31.74.195:42721)
I20260806 02:59:27.505981 5358 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.74.195@KRBTEST.COM'} at 127.31.74.195:44013
I20260806 02:59:27.510104 32043 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/krb5kdc/test-admin.keytab.
Aug 06 02:59:27 dist-test-slave-dhph krb5kdc[5312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:59:27.559077 32043 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260806 02:59:27.565766 5835 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:59:27.562768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47340 (local address 127.31.74.193:40647)
0806 02:59:27.563446 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:27.563453 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:27.563481 (+ 28us) server_negotiation.cc:407] Connection header received
0806 02:59:27.563545 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:27.563552 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:27.563807 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:27.563940 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:27.565309 (+ 1369us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47340: BlockingRecv error: recv got EOF from 127.0.0.1:47340 (error 108)
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":350,"threads_started":1}
W20260806 02:59:27.565754 32043 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/krb5kdc/krb5cc' of type FILE
Aug 06 02:59:27 dist-test-slave-dhph krb5kdc[5312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:59:27.568141 32043 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 06 02:59:27 dist-test-slave-dhph krb5kdc[5312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:59:27.569999 32043 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 06 02:59:27 dist-test-slave-dhph krb5kdc[5312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.74.193@KRBTEST.COM
I20260806 02:59:27.592587 5835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:27.572285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47342 (local address 127.31.74.193:40647)
0806 02:59:27.572521 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:27.572527 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:27.572666 (+ 139us) server_negotiation.cc:407] Connection header received
0806 02:59:27.573224 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:27.573228 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:27.573302 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:27.573395 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:27.575447 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:27.576768 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:27.578787 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:27.579631 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:27.579761 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:27.585134 (+ 5373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:27.585186 (+ 52us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:27.585208 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:27.585273 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:27.588655 (+ 3382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:27.589641 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:27.589651 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:27.589653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:27.589812 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:27.590298 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:27.590303 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:27.590306 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:27.591181 (+ 875us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:27.591499 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:27.591912 (+ 413us) server_negotiation.cc:299] Negotiation successful
0806 02:59:27.592281 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
W20260806 02:59:27.602082 32043 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/krb5kdc/krb5cc' of type FILE
W20260806 02:59:27.602291 5835 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:59:27.598444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47348 (local address 127.31.74.193:40647)
0806 02:59:27.598727 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:27.598734 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:27.599327 (+ 593us) server_negotiation.cc:407] Connection header received
0806 02:59:27.599614 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:27.599622 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:27.599774 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:27.599918 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:27.601960 (+ 2042us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47348: BlockingRecv error: recv got EOF from 127.0.0.1:47348 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
Aug 06 02:59:27 dist-test-slave-dhph krb5kdc[5312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:59:27.604525 32043 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 06 02:59:27 dist-test-slave-dhph krb5kdc[5312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:59:27.606474 32043 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 06 02:59:27 dist-test-slave-dhph krb5kdc[5312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.74.193@KRBTEST.COM
I20260806 02:59:27.631242 5835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:27.609938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47352 (local address 127.31.74.193:40647)
0806 02:59:27.610420 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:27.610618 (+ 198us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:27.610656 (+ 38us) server_negotiation.cc:407] Connection header received
0806 02:59:27.610861 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:27.610868 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:27.610965 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:27.611031 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:27.613971 (+ 2940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:27.615096 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:27.616654 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:27.617139 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:27.617321 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:27.624547 (+ 7226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:27.624567 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:27.624572 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:27.624639 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:27.628133 (+ 3494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:27.628939 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:27.628945 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:27.628948 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:27.629107 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:27.629526 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:27.629531 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:27.629534 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:27.629842 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:27.630145 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:27.630494 (+ 349us) server_negotiation.cc:299] Negotiation successful
0806 02:59:27.630765 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317}
W20260806 02:59:27.637888 5835 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:59:27.635595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47354 (local address 127.31.74.193:40647)
0806 02:59:27.635875 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:27.635880 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:27.635898 (+ 18us) server_negotiation.cc:407] Connection header received
0806 02:59:27.635951 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:27.635955 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:27.636031 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:27.636127 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:27.637564 (+ 1437us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47354: BlockingRecv error: recv got EOF from 127.0.0.1:47354 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
W20260806 02:59:27.639055 32043 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785985105255475-32043-0/krb5kdc/krb5cc' of type FILE
Aug 06 02:59:27 dist-test-slave-dhph krb5kdc[5312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:59:27.642937 32043 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 06 02:59:27 dist-test-slave-dhph krb5kdc[5312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:59:27.645078 32043 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 06 02:59:27 dist-test-slave-dhph krb5kdc[5312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.74.193@KRBTEST.COM
I20260806 02:59:27.667079 5835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:27.647139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47364 (local address 127.31.74.193:40647)
0806 02:59:27.647526 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:27.647536 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:27.647849 (+ 313us) server_negotiation.cc:407] Connection header received
0806 02:59:27.648066 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:27.648071 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:27.648140 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:27.648249 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:27.649321 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:27.650065 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:27.651512 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:27.651947 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:27.652168 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:27.657083 (+ 4915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:27.657092 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:27.657096 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:27.657160 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:27.662347 (+ 5187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:27.663767 (+ 1420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:27.663774 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:27.663776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:27.663997 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:27.664957 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:27.665073 (+ 116us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:27.665076 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:27.665466 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:27.665753 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:27.666407 (+ 654us) server_negotiation.cc:299] Negotiation successful
0806 02:59:27.666788 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260806 02:59:27.669917 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 5420
I20260806 02:59:27.688886 5835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:27.670895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47370 (local address 127.31.74.193:40647)
0806 02:59:27.671138 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:27.671145 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:27.671385 (+ 240us) server_negotiation.cc:407] Connection header received
0806 02:59:27.671634 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:27.671639 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:27.671713 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:27.671864 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:27.673769 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:27.674496 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:27.676647 (+ 2151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:27.677181 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:27.677310 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:27.681020 (+ 3710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:27.681027 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:27.681032 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:27.681080 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:27.684702 (+ 3622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:27.685806 (+ 1104us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:27.685811 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:27.685813 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:27.685973 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:27.686651 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:27.686661 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:27.686666 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:27.686988 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:27.687239 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:27.687729 (+ 490us) server_negotiation.cc:299] Negotiation successful
0806 02:59:27.688297 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260806 02:59:27.689673 5545 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:59:27.796012 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 5420
I20260806 02:59:27.819334 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 5556
I20260806 02:59:27.828357 5689 heartbeater.cc:503] Master 127.31.74.254:40053 was elected leader, sending a full tablet report...
Aug 06 02:59:27 dist-test-slave-dhph krb5kdc[5312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.74.194@KRBTEST.COM
I20260806 02:59:27.842686 5848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:27.820556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35090 (local address 127.31.74.194:37359)
0806 02:59:27.821286 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:27.821294 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:27.821324 (+ 30us) server_negotiation.cc:407] Connection header received
0806 02:59:27.821393 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:27.821399 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:27.821627 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:27.822159 (+ 532us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:27.823447 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:27.824935 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:27.826554 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:27.827750 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:27.827928 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:27.834358 (+ 6430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:27.834378 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:27.834395 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:27.834456 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:27.838932 (+ 4476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:27.839954 (+ 1022us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:27.839973 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:27.839976 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:27.840132 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:27.840594 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:27.840600 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:27.840603 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:27.841044 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:27.841413 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:27.841804 (+ 391us) server_negotiation.cc:299] Negotiation successful
0806 02:59:27.842140 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":459,"threads_started":1}
I20260806 02:59:27.844122 5684 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:59:27.947839 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 5556
I20260806 02:59:27.969405 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 5695
Aug 06 02:59:27 dist-test-slave-dhph krb5kdc[5312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.74.195@KRBTEST.COM
I20260806 02:59:27.991468 5850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:27.970876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57978 (local address 127.31.74.195:42721)
0806 02:59:27.971772 (+ 896us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:27.971782 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:27.971830 (+ 48us) server_negotiation.cc:407] Connection header received
0806 02:59:27.971920 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:27.971928 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:27.972236 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:27.972696 (+ 460us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:27.973907 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:27.975026 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:27.976419 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:27.977005 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:27.977163 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:27.982945 (+ 5782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:27.982966 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:27.982983 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:27.983043 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:27.987130 (+ 4087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:27.988400 (+ 1270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:27.988408 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:27.988411 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:27.988585 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:27.989296 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:27.989300 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:27.989303 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:27.989771 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:27.990045 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:27.990518 (+ 473us) server_negotiation.cc:299] Negotiation successful
0806 02:59:27.991008 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":410,"threads_started":1}
I20260806 02:59:27.993299 5821 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:59:28.087100 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 5695
I20260806 02:59:28.108244 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 5325
I20260806 02:59:28.110373 5388 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:59:28.224208 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 5325
2026-08-06T02:59:28Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4935 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:59:28 dist-test-slave-dhph krb5kdc[5855](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:59:28 dist-test-slave-dhph krb5kdc[5855](info): set up 2 sockets
Aug 06 02:59:28 dist-test-slave-dhph krb5kdc[5855](info): commencing operation
krb5kdc: starting...
W20260806 02:59:29.469501 32043 mini_kdc.cc:121] Time spent starting KDC: real 1.199s user 0.000s sys 0.046s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/krb5kdc/test-user.keytab.
Aug 06 02:59:29 dist-test-slave-dhph krb5kdc[5855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985169, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:59:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:59:29Z Disabled control of system clock
WARNING: no policy specified for oryx/127.31.74.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.74.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.74.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/krb5kdc/oryx_127.31.74.254.keytab.
Entry for principal oryx/127.31.74.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/krb5kdc/oryx_127.31.74.254.keytab.
I20260806 02:59:29.783493 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:36051
--webserver_interface=127.31.74.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/krb5kdc/oryx_127.31.74.254.keytab
--principal=oryx/127.31.74.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:40361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.74.254:36051
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:59:30.172336 5868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:59:30.172884 5868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:59:30.173178 5868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:59:30.181464 5868 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:59:30.181617 5868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:59:30.181666 5868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:59:30.181795 5868 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:59:30.181875 5868 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:59:30.193970 5868 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:40361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.74.254:36051
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/krb5kdc/oryx_127.31.74.254.keytab
--principal=oryx/127.31.74.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:36051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.74.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:59:30.195593 5868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:59:30.197077 5868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:59:30.212108 5873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:30.212169 5876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:30.212270 5874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:59:30.212953 5868 server_base.cc:1034] running on GCE node
I20260806 02:59:30.214202 5868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:59:30.216136 5868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:59:30.217427 5868 hybrid_clock.cc:648] HybridClock initialized: now 1785985170217353 us; error 75 us; skew 500 ppm
I20260806 02:59:30.217867 5868 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:59:30 dist-test-slave-dhph krb5kdc[5855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985170, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.74.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:59:30.222187 5868 init.cc:377] Logged in from keytab as oryx/127.31.74.254@KRBTEST.COM (short username oryx)
I20260806 02:59:30.224579 5868 webserver.cc:460] Webserver started at http://127.31.74.254:34587/ using document root <none> and password file <none>
I20260806 02:59:30.225332 5868 fs_manager.cc:359] Metadata directory not provided
I20260806 02:59:30.225453 5868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:59:30.225747 5868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:59:30.229060 5868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/master-0/data/instance:
uuid: "ec7ce3806c9d48a2bbbfacaed8423300"
format_stamp: "Formatted at 2026-08-06 02:59:30 on dist-test-slave-dhph"
server_key: "58633b854762b7a3dcdb6a5409407741"
server_key_iv: "f03375a80dc220cc84a779bb9737cb24"
server_key_version: "encryptionkey@0"
I20260806 02:59:30.229885 5868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/master-0/wal/instance:
uuid: "ec7ce3806c9d48a2bbbfacaed8423300"
format_stamp: "Formatted at 2026-08-06 02:59:30 on dist-test-slave-dhph"
server_key: "58633b854762b7a3dcdb6a5409407741"
server_key_iv: "f03375a80dc220cc84a779bb9737cb24"
server_key_version: "encryptionkey@0"
I20260806 02:59:30.235982 5868 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260806 02:59:30.241204 5884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:30.243100 5868 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260806 02:59:30.243351 5868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/master-0/wal
uuid: "ec7ce3806c9d48a2bbbfacaed8423300"
format_stamp: "Formatted at 2026-08-06 02:59:30 on dist-test-slave-dhph"
server_key: "58633b854762b7a3dcdb6a5409407741"
server_key_iv: "f03375a80dc220cc84a779bb9737cb24"
server_key_version: "encryptionkey@0"
I20260806 02:59:30.243603 5868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:59:30.270865 5868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:59:30.276026 5868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:59:30.276347 5868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:59:30.300890 5868 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.254:36051
I20260806 02:59:30.300989 5938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.254:36051 every 8 connection(s)
I20260806 02:59:30.302448 5868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
I20260806 02:59:30.305598 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 5868
I20260806 02:59:30.305894 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/master-0/wal/instance
I20260806 02:59:30.306738 32043 external_mini_cluster.cc:1428] Setting key 724911af6d489d89f6f1407e236a5d6b
I20260806 02:59:30.306947 5939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:59:30.323678 5939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec7ce3806c9d48a2bbbfacaed8423300: Bootstrap starting.
Aug 06 02:59:30 dist-test-slave-dhph krb5kdc[5855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985169, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM
I20260806 02:59:30.327098 5939 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec7ce3806c9d48a2bbbfacaed8423300: Neither blocks nor log segments found. Creating new log.
I20260806 02:59:30.328960 5939 log.cc:824] T 00000000000000000000000000000000 P ec7ce3806c9d48a2bbbfacaed8423300: Log is configured to *not* fsync() on all Append() calls
I20260806 02:59:30.333086 5939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec7ce3806c9d48a2bbbfacaed8423300: No bootstrap required, opened a new log
I20260806 02:59:30.337018 5942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:30.309035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46262 (local address 127.31.74.254:36051)
0806 02:59:30.309945 (+ 910us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:30.309961 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:30.310008 (+ 47us) server_negotiation.cc:407] Connection header received
0806 02:59:30.312618 (+ 2610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:30.312649 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:30.313144 (+ 495us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:30.313538 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:30.316391 (+ 2853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:30.318163 (+ 1772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:30.320335 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:30.321368 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:30.321611 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:30.327645 (+ 6034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:30.327654 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:30.327671 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:30.327722 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:30.331942 (+ 4220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:30.332835 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:30.332843 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:30.332849 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:30.333034 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:30.333537 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:30.333542 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:30.333544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:30.334121 (+ 577us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:30.334514 (+ 393us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:30.335013 (+ 499us) server_negotiation.cc:299] Negotiation successful
0806 02:59:30.335530 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":689,"thread_start_us":447,"threads_started":1}
I20260806 02:59:30.342988 5939 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec7ce3806c9d48a2bbbfacaed8423300 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec7ce3806c9d48a2bbbfacaed8423300" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36051 } }
I20260806 02:59:30.343360 5939 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec7ce3806c9d48a2bbbfacaed8423300 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:59:30.343425 5939 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec7ce3806c9d48a2bbbfacaed8423300 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec7ce3806c9d48a2bbbfacaed8423300, State: Initialized, Role: FOLLOWER
I20260806 02:59:30.343928 5939 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec7ce3806c9d48a2bbbfacaed8423300 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec7ce3806c9d48a2bbbfacaed8423300" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36051 } }
I20260806 02:59:30.344151 5939 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec7ce3806c9d48a2bbbfacaed8423300 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:59:30.344244 5939 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec7ce3806c9d48a2bbbfacaed8423300 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:59:30.344431 5939 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec7ce3806c9d48a2bbbfacaed8423300 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:59:30.347142 5939 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec7ce3806c9d48a2bbbfacaed8423300 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec7ce3806c9d48a2bbbfacaed8423300" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36051 } }
I20260806 02:59:30.347891 5939 leader_election.cc:302] T 00000000000000000000000000000000 P ec7ce3806c9d48a2bbbfacaed8423300 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec7ce3806c9d48a2bbbfacaed8423300; no voters:
I20260806 02:59:30.348708 5939 leader_election.cc:288] T 00000000000000000000000000000000 P ec7ce3806c9d48a2bbbfacaed8423300 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:59:30.348927 5944 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec7ce3806c9d48a2bbbfacaed8423300 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:59:30.350358 5944 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec7ce3806c9d48a2bbbfacaed8423300 [term 1 LEADER]: Becoming Leader. State: Replica: ec7ce3806c9d48a2bbbfacaed8423300, State: Running, Role: LEADER
I20260806 02:59:30.350997 5944 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec7ce3806c9d48a2bbbfacaed8423300 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec7ce3806c9d48a2bbbfacaed8423300" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36051 } }
I20260806 02:59:30.351326 5939 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec7ce3806c9d48a2bbbfacaed8423300 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:59:30.354561 5946 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec7ce3806c9d48a2bbbfacaed8423300 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec7ce3806c9d48a2bbbfacaed8423300. Latest consensus state: current_term: 1 leader_uuid: "ec7ce3806c9d48a2bbbfacaed8423300" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec7ce3806c9d48a2bbbfacaed8423300" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36051 } } }
I20260806 02:59:30.355055 5946 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec7ce3806c9d48a2bbbfacaed8423300 [sys.catalog]: This master's current role is: LEADER
I20260806 02:59:30.355610 5945 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec7ce3806c9d48a2bbbfacaed8423300 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec7ce3806c9d48a2bbbfacaed8423300" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec7ce3806c9d48a2bbbfacaed8423300" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36051 } } }
I20260806 02:59:30.355964 5945 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec7ce3806c9d48a2bbbfacaed8423300 [sys.catalog]: This master's current role is: LEADER
I20260806 02:59:30.357999 5953 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:59:30.364446 5953 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:59:30.373966 5953 catalog_manager.cc:1347] Generated new cluster ID: 8e6a631af4aa433eb774f4779a94b441
I20260806 02:59:30.374118 5953 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:59:30.398206 5953 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:59:30.399840 5953 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:59:30.417801 5953 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec7ce3806c9d48a2bbbfacaed8423300: Generated new TSK 0
I20260806 02:59:30.418877 5953 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.31.74.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.74.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.74.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/krb5kdc/oryx_127.31.74.193.keytab.
Entry for principal oryx/127.31.74.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/krb5kdc/oryx_127.31.74.193.keytab.
I20260806 02:59:30.491446 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.193:0
--local_ip_for_outbound_sockets=127.31.74.193
--webserver_interface=127.31.74.193
--webserver_port=0
--tserver_master_addrs=127.31.74.254:36051
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/krb5kdc/oryx_127.31.74.193.keytab
--principal=oryx/127.31.74.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:40361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:59:30.876065 5965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:59:30.876526 5965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:59:30.876757 5965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:59:30.884541 5965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:59:30.884671 5965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:59:30.884848 5965 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.193
I20260806 02:59:30.895164 5965 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:40361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/krb5kdc/oryx_127.31.74.193.keytab
--principal=oryx/127.31.74.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.74.193
--webserver_port=0
--tserver_master_addrs=127.31.74.254:36051
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.193
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:59:30.896579 5965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:59:30.897912 5965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:59:30.912839 5971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:30.912885 5970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:30.913626 5973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:59:30.914319 5965 server_base.cc:1034] running on GCE node
I20260806 02:59:30.915273 5965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:59:30.916363 5965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:59:30.917683 5965 hybrid_clock.cc:648] HybridClock initialized: now 1785985170917595 us; error 82 us; skew 500 ppm
I20260806 02:59:30.918062 5965 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:59:30 dist-test-slave-dhph krb5kdc[5855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985170, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.74.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:59:30.922458 5965 init.cc:377] Logged in from keytab as oryx/127.31.74.193@KRBTEST.COM (short username oryx)
I20260806 02:59:30.924291 5965 webserver.cc:460] Webserver started at http://127.31.74.193:41173/ using document root <none> and password file <none>
I20260806 02:59:30.924997 5965 fs_manager.cc:359] Metadata directory not provided
I20260806 02:59:30.925104 5965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:59:30.925369 5965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:59:30.928535 5965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-0/data/instance:
uuid: "140e7485da14480cb28af227351c865f"
format_stamp: "Formatted at 2026-08-06 02:59:30 on dist-test-slave-dhph"
server_key: "85871f41c779d879a68629bf52bc0247"
server_key_iv: "63c4b46959f031926a7f66f90a3816d1"
server_key_version: "encryptionkey@0"
I20260806 02:59:30.929317 5965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-0/wal/instance:
uuid: "140e7485da14480cb28af227351c865f"
format_stamp: "Formatted at 2026-08-06 02:59:30 on dist-test-slave-dhph"
server_key: "85871f41c779d879a68629bf52bc0247"
server_key_iv: "63c4b46959f031926a7f66f90a3816d1"
server_key_version: "encryptionkey@0"
I20260806 02:59:30.934891 5965 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260806 02:59:30.938521 5981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:30.940006 5965 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:59:30.940147 5965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-0/wal
uuid: "140e7485da14480cb28af227351c865f"
format_stamp: "Formatted at 2026-08-06 02:59:30 on dist-test-slave-dhph"
server_key: "85871f41c779d879a68629bf52bc0247"
server_key_iv: "63c4b46959f031926a7f66f90a3816d1"
server_key_version: "encryptionkey@0"
I20260806 02:59:30.940325 5965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:59:30.955276 5965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:59:30.959774 5965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:59:30.960052 5965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:59:30.961069 5965 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:59:30.963073 5965 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:59:30.963162 5965 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:30.963266 5965 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:59:30.963325 5965 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:31.001389 5965 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.193:46705
I20260806 02:59:31.001451 6095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.193:46705 every 8 connection(s)
I20260806 02:59:31.002897 5965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
I20260806 02:59:31.005908 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 5965
I20260806 02:59:31.006196 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-0/wal/instance
I20260806 02:59:31.007126 32043 external_mini_cluster.cc:1428] Setting key afad356bed53f2538cac03957896286d
Aug 06 02:59:31 dist-test-slave-dhph krb5kdc[5855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985170, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.74.193@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM
I20260806 02:59:31.030202 6098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:31.006691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.193:58093 (local address 127.31.74.254:36051)
0806 02:59:31.009167 (+ 2476us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:31.009173 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:31.009196 (+ 23us) server_negotiation.cc:407] Connection header received
0806 02:59:31.011305 (+ 2109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:31.011314 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:31.011416 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:31.011544 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:31.013358 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:31.014529 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:31.016307 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:31.017028 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:31.017190 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:31.022137 (+ 4947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:31.022152 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:31.022157 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:31.022203 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:31.025284 (+ 3081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:31.026485 (+ 1201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:31.026490 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:31.026493 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:31.026720 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:31.027320 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:31.027324 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:31.027327 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:31.027670 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:31.027874 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:31.029622 (+ 1748us) server_negotiation.cc:299] Negotiation successful
0806 02:59:31.029896 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2359,"thread_start_us":2249,"threads_started":1}
I20260806 02:59:31.032060 6099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:31.007405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:36051 (local address 127.31.74.193:58093)
0806 02:59:31.008636 (+ 1231us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:31.008708 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:31.010999 (+ 2291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:31.011882 (+ 883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:31.011892 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:31.012484 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:59:31.013129 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:31.013149 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:31.014752 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:31.014756 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:31.016119 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:31.016134 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:31.017341 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:31.017347 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:31.017603 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:31.018459 (+ 856us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:59:31.018486 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:59:31.021450 (+ 2964us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:59:31.025492 (+ 4042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:31.025507 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:31.025530 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:31.026287 (+ 757us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:31.026855 (+ 568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:31.026862 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:31.026868 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:31.027161 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:31.028094 (+ 933us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:59:31.028111 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:59:31.029019 (+ 908us) client_negotiation.cc:769] Sending connection context
0806 02:59:31.029487 (+ 468us) client_negotiation.cc:240] Negotiation successful
0806 02:59:31.030236 (+ 749us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":777,"thread_start_us":458,"threads_started":1}
I20260806 02:59:31.034024 6096 heartbeater.cc:347] Connected to a master server at 127.31.74.254:36051
I20260806 02:59:31.034359 6096 heartbeater.cc:464] Registering TS with master...
I20260806 02:59:31.035218 6096 heartbeater.cc:511] Master 127.31.74.254:36051 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.31.74.194@KRBTEST.COM; defaulting to no policy
I20260806 02:59:31.037712 5904 ts_manager.cc:194] Registered new tserver with Master: 140e7485da14480cb28af227351c865f (127.31.74.193:46705)
I20260806 02:59:31.039417 5904 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.74.193@KRBTEST.COM'} at 127.31.74.193:58093
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.74.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.74.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/krb5kdc/oryx_127.31.74.194.keytab.
Entry for principal oryx/127.31.74.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/krb5kdc/oryx_127.31.74.194.keytab.
I20260806 02:59:31.079653 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.194:0
--local_ip_for_outbound_sockets=127.31.74.194
--webserver_interface=127.31.74.194
--webserver_port=0
--tserver_master_addrs=127.31.74.254:36051
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/krb5kdc/oryx_127.31.74.194.keytab
--principal=oryx/127.31.74.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:40361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:59:31.461790 6102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:59:31.462145 6102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:59:31.462266 6102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:59:31.471446 6102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:59:31.471607 6102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:59:31.471885 6102 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.194
I20260806 02:59:31.483527 6102 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:40361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/krb5kdc/oryx_127.31.74.194.keytab
--principal=oryx/127.31.74.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.74.194
--webserver_port=0
--tserver_master_addrs=127.31.74.254:36051
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.194
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:59:31.484956 6102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:59:31.486464 6102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:59:31.502249 6108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:31.502301 6107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:31.502328 6110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:59:31.503187 6102 server_base.cc:1034] running on GCE node
I20260806 02:59:31.503930 6102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:59:31.505054 6102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:59:31.506408 6102 hybrid_clock.cc:648] HybridClock initialized: now 1785985171506308 us; error 59 us; skew 500 ppm
I20260806 02:59:31.506855 6102 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:59:31 dist-test-slave-dhph krb5kdc[5855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985171, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.74.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:59:31.510416 6102 init.cc:377] Logged in from keytab as oryx/127.31.74.194@KRBTEST.COM (short username oryx)
I20260806 02:59:31.512084 6102 webserver.cc:460] Webserver started at http://127.31.74.194:46785/ using document root <none> and password file <none>
I20260806 02:59:31.512880 6102 fs_manager.cc:359] Metadata directory not provided
I20260806 02:59:31.513046 6102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:59:31.513384 6102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:59:31.516697 6102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-1/data/instance:
uuid: "d45fe0ed8f6b4ddb9b3f31f14ba18ace"
format_stamp: "Formatted at 2026-08-06 02:59:31 on dist-test-slave-dhph"
server_key: "164e3c0eaf557d7fcd84b519269ce02b"
server_key_iv: "b8cc74321680930a85b35a3aaf6a1740"
server_key_version: "encryptionkey@0"
I20260806 02:59:31.517570 6102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-1/wal/instance:
uuid: "d45fe0ed8f6b4ddb9b3f31f14ba18ace"
format_stamp: "Formatted at 2026-08-06 02:59:31 on dist-test-slave-dhph"
server_key: "164e3c0eaf557d7fcd84b519269ce02b"
server_key_iv: "b8cc74321680930a85b35a3aaf6a1740"
server_key_version: "encryptionkey@0"
I20260806 02:59:31.525900 6102 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.008s sys 0.000s
I20260806 02:59:31.530373 6117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:31.532294 6102 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260806 02:59:31.532541 6102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-1/wal
uuid: "d45fe0ed8f6b4ddb9b3f31f14ba18ace"
format_stamp: "Formatted at 2026-08-06 02:59:31 on dist-test-slave-dhph"
server_key: "164e3c0eaf557d7fcd84b519269ce02b"
server_key_iv: "b8cc74321680930a85b35a3aaf6a1740"
server_key_version: "encryptionkey@0"
I20260806 02:59:31.532800 6102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:59:31.558876 6102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:59:31.564143 6102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:59:31.564474 6102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:59:31.565608 6102 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:59:31.567169 6102 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:59:31.567284 6102 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:31.567440 6102 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:59:31.567515 6102 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:31.610679 6102 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.194:41805
I20260806 02:59:31.611079 6232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.194:41805 every 8 connection(s)
I20260806 02:59:31.612274 6102 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
I20260806 02:59:31.622921 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 6102
I20260806 02:59:31.623150 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-1/wal/instance
I20260806 02:59:31.623682 32043 external_mini_cluster.cc:1428] Setting key 3c641624857f5755e7ae9f330cb6ca01
Aug 06 02:59:31 dist-test-slave-dhph krb5kdc[5855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985171, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.74.194@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM
I20260806 02:59:31.648662 6235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:31.616546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.194:59155 (local address 127.31.74.254:36051)
0806 02:59:31.617282 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:31.617290 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:31.618569 (+ 1279us) server_negotiation.cc:407] Connection header received
0806 02:59:31.620701 (+ 2132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:31.620707 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:31.620806 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:31.620928 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:31.626970 (+ 6042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:31.628243 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:31.630432 (+ 2189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:31.631383 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:31.632031 (+ 648us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:31.637941 (+ 5910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:31.637952 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:31.637960 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:31.638016 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:31.642409 (+ 4393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:31.644621 (+ 2212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:31.644626 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:31.644628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:31.645051 (+ 423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:31.645933 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:31.645949 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:31.645951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:31.646396 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:31.646544 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:31.647997 (+ 1453us) server_negotiation.cc:299] Negotiation successful
0806 02:59:31.648342 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":406,"threads_started":1}
I20260806 02:59:31.650687 6236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:31.617129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:36051 (local address 127.31.74.194:59155)
0806 02:59:31.618351 (+ 1222us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:31.618406 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:31.620339 (+ 1933us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:31.622437 (+ 2098us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:31.622449 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:31.623521 (+ 1072us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:59:31.626256 (+ 2735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:31.626290 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:31.628550 (+ 2260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:31.628556 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:31.630128 (+ 1572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:31.630148 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:31.632114 (+ 1966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:31.632121 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:31.632673 (+ 552us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:31.633659 (+ 986us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:59:31.633691 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:59:31.637079 (+ 3388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:59:31.642929 (+ 5850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:31.642946 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:31.642970 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:31.644394 (+ 1424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:31.645331 (+ 937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:31.645340 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:31.645347 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:31.645702 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:31.646643 (+ 941us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:59:31.646657 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:59:31.647290 (+ 633us) client_negotiation.cc:769] Sending connection context
0806 02:59:31.647884 (+ 594us) client_negotiation.cc:240] Negotiation successful
0806 02:59:31.648625 (+ 741us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":821,"thread_start_us":518,"threads_started":1}
I20260806 02:59:31.653388 6233 heartbeater.cc:347] Connected to a master server at 127.31.74.254:36051
I20260806 02:59:31.653841 6233 heartbeater.cc:464] Registering TS with master...
I20260806 02:59:31.654805 6233 heartbeater.cc:511] Master 127.31.74.254:36051 requested a full tablet report, sending...
I20260806 02:59:31.657037 5904 ts_manager.cc:194] Registered new tserver with Master: d45fe0ed8f6b4ddb9b3f31f14ba18ace (127.31.74.194:41805)
I20260806 02:59:31.658113 5904 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.74.194@KRBTEST.COM'} at 127.31.74.194:59155
WARNING: no policy specified for oryx/127.31.74.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.74.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.74.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/krb5kdc/oryx_127.31.74.195.keytab.
Entry for principal oryx/127.31.74.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/krb5kdc/oryx_127.31.74.195.keytab.
I20260806 02:59:31.711820 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.195:0
--local_ip_for_outbound_sockets=127.31.74.195
--webserver_interface=127.31.74.195
--webserver_port=0
--tserver_master_addrs=127.31.74.254:36051
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/krb5kdc/oryx_127.31.74.195.keytab
--principal=oryx/127.31.74.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:40361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:59:32.042958 6096 heartbeater.cc:503] Master 127.31.74.254:36051 was elected leader, sending a full tablet report...
W20260806 02:59:32.118877 6239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:59:32.119204 6239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:59:32.119307 6239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:59:32.128197 6239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:59:32.128372 6239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:59:32.128501 6239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.195
I20260806 02:59:32.142519 6239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:40361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/krb5kdc/oryx_127.31.74.195.keytab
--principal=oryx/127.31.74.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.74.195
--webserver_port=0
--tserver_master_addrs=127.31.74.254:36051
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.195
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:59:32.144054 6239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:59:32.145555 6239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:59:32.163097 6247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:32.164400 6244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:32.164510 6245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:59:32.165587 6239 server_base.cc:1034] running on GCE node
I20260806 02:59:32.166834 6239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:59:32.168853 6239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:59:32.170214 6239 hybrid_clock.cc:648] HybridClock initialized: now 1785985172170118 us; error 101 us; skew 500 ppm
I20260806 02:59:32.170768 6239 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:59:32 dist-test-slave-dhph krb5kdc[5855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985172, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.74.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:59:32.178747 6239 init.cc:377] Logged in from keytab as oryx/127.31.74.195@KRBTEST.COM (short username oryx)
I20260806 02:59:32.180440 6239 webserver.cc:460] Webserver started at http://127.31.74.195:35761/ using document root <none> and password file <none>
I20260806 02:59:32.181197 6239 fs_manager.cc:359] Metadata directory not provided
I20260806 02:59:32.181313 6239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:59:32.181584 6239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:59:32.184887 6239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-2/data/instance:
uuid: "7ed1c497e3454a6eb54f4bcfda8552de"
format_stamp: "Formatted at 2026-08-06 02:59:32 on dist-test-slave-dhph"
server_key: "880492327dfab0c8727b0d7726782321"
server_key_iv: "8458138521a92c973026a2198a69bb5f"
server_key_version: "encryptionkey@0"
I20260806 02:59:32.185811 6239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-2/wal/instance:
uuid: "7ed1c497e3454a6eb54f4bcfda8552de"
format_stamp: "Formatted at 2026-08-06 02:59:32 on dist-test-slave-dhph"
server_key: "880492327dfab0c8727b0d7726782321"
server_key_iv: "8458138521a92c973026a2198a69bb5f"
server_key_version: "encryptionkey@0"
I20260806 02:59:32.191766 6239 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260806 02:59:32.195909 6254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:32.199126 6239 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.001s
I20260806 02:59:32.199363 6239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-2/wal
uuid: "7ed1c497e3454a6eb54f4bcfda8552de"
format_stamp: "Formatted at 2026-08-06 02:59:32 on dist-test-slave-dhph"
server_key: "880492327dfab0c8727b0d7726782321"
server_key_iv: "8458138521a92c973026a2198a69bb5f"
server_key_version: "encryptionkey@0"
I20260806 02:59:32.199818 6239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:59:32.224702 6239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:59:32.230425 6239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:59:32.230865 6239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:59:32.232107 6239 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:59:32.233935 6239 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:59:32.234040 6239 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:32.234198 6239 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:59:32.234259 6239 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:32.277134 6239 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.195:41485
I20260806 02:59:32.277259 6370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.195:41485 every 8 connection(s)
I20260806 02:59:32.278455 6239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
I20260806 02:59:32.287883 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 6239
I20260806 02:59:32.288067 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-2/wal/instance
I20260806 02:59:32.288622 32043 external_mini_cluster.cc:1428] Setting key a22eb81857d09ae25851275d0c52090b
Aug 06 02:59:32 dist-test-slave-dhph krb5kdc[5855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985172, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.74.195@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM
I20260806 02:59:32.302726 6373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:32.281168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.195:41439 (local address 127.31.74.254:36051)
0806 02:59:32.281693 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:32.281709 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:32.282975 (+ 1266us) server_negotiation.cc:407] Connection header received
0806 02:59:32.285036 (+ 2061us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:32.285040 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:32.285126 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:32.285231 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:32.287064 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:32.288217 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:32.290390 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:32.291020 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:32.291207 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:32.295371 (+ 4164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:32.295376 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:32.295379 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:32.295417 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:32.299017 (+ 3600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:32.300043 (+ 1026us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:32.300053 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:32.300056 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:32.300215 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:32.300785 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:32.300790 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:32.300792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:32.301103 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:32.301270 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:32.302014 (+ 744us) server_negotiation.cc:299] Negotiation successful
0806 02:59:32.302278 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":253,"threads_started":1}
I20260806 02:59:32.304270 6374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:32.281807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:36051 (local address 127.31.74.195:41439)
0806 02:59:32.282757 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:32.282812 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:32.284747 (+ 1935us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:32.285498 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:32.285507 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:32.286059 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:59:32.286864 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:32.286888 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:32.288408 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:32.288413 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:32.290214 (+ 1801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:32.290229 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:32.291343 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:32.291353 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:32.291647 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:32.292466 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:59:32.292498 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:59:32.295092 (+ 2594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:59:32.299221 (+ 4129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:32.299232 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:32.299257 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:32.299902 (+ 645us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:32.300376 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:32.300380 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:32.300384 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:32.300654 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:32.301240 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:59:32.301248 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:59:32.301637 (+ 389us) client_negotiation.cc:769] Sending connection context
0806 02:59:32.301917 (+ 280us) client_negotiation.cc:240] Negotiation successful
0806 02:59:32.302507 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":296,"threads_started":1}
I20260806 02:59:32.306002 6371 heartbeater.cc:347] Connected to a master server at 127.31.74.254:36051
I20260806 02:59:32.306376 6371 heartbeater.cc:464] Registering TS with master...
I20260806 02:59:32.307310 6371 heartbeater.cc:511] Master 127.31.74.254:36051 requested a full tablet report, sending...
I20260806 02:59:32.309437 5904 ts_manager.cc:194] Registered new tserver with Master: 7ed1c497e3454a6eb54f4bcfda8552de (127.31.74.195:41485)
I20260806 02:59:32.310349 5904 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.74.195@KRBTEST.COM'} at 127.31.74.195:41439
I20260806 02:59:32.318737 32043 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.31.74.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.74.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.74.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/krb5kdc/oryx_127.31.74.253.keytab.
Entry for principal oryx/127.31.74.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/krb5kdc/oryx_127.31.74.253.keytab.
I20260806 02:59:32.404868 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.253:40869
--webserver_interface=127.31.74.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/krb5kdc/oryx_127.31.74.253.keytab
--principal=oryx/127.31.74.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:40361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.74.254:36051,127.31.74.253:40869
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:59:32.662037 6233 heartbeater.cc:503] Master 127.31.74.254:36051 was elected leader, sending a full tablet report...
W20260806 02:59:32.820863 6377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:59:32.821269 6377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:59:32.821465 6377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:59:32.830850 6377 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:59:32.831060 6377 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:59:32.831177 6377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:59:32.831287 6377 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:59:32.831368 6377 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:59:32.843154 6377 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:40361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.31.74.254:36051,127.31.74.253:40869
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/krb5kdc/oryx_127.31.74.253.keytab
--principal=oryx/127.31.74.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.253:40869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.31.74.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
W20260806 02:59:32.843668 6377 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.31.74.254:36051,127.31.74.253:40869'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260806 02:59:32.844987 6377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:59:32.846575 6377 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:59:32.861539 6385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:32.862282 6382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:32.862308 6383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:59:32.862660 6377 server_base.cc:1034] running on GCE node
I20260806 02:59:32.863621 6377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:59:32.865298 6377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:59:32.866997 6377 hybrid_clock.cc:648] HybridClock initialized: now 1785985172866805 us; error 153 us; skew 500 ppm
I20260806 02:59:32.867677 6377 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:59:32 dist-test-slave-dhph krb5kdc[5855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985172, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.74.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:59:32.876891 6377 init.cc:377] Logged in from keytab as oryx/127.31.74.253@KRBTEST.COM (short username oryx)
I20260806 02:59:32.879125 6377 webserver.cc:460] Webserver started at http://127.31.74.253:34395/ using document root <none> and password file <none>
I20260806 02:59:32.880057 6377 fs_manager.cc:359] Metadata directory not provided
I20260806 02:59:32.880169 6377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:59:32.880486 6377 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:59:32.884081 6377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/master-1/data/instance:
uuid: "001d2d57ebfd4d6aa7d81fde49a50231"
format_stamp: "Formatted at 2026-08-06 02:59:32 on dist-test-slave-dhph"
server_key: "f316790adaa744ecd098fc7f2e399fad"
server_key_iv: "5e8365f3ea36f56019bb639372d8c7c8"
server_key_version: "encryptionkey@0"
I20260806 02:59:32.885130 6377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/master-1/wal/instance:
uuid: "001d2d57ebfd4d6aa7d81fde49a50231"
format_stamp: "Formatted at 2026-08-06 02:59:32 on dist-test-slave-dhph"
server_key: "f316790adaa744ecd098fc7f2e399fad"
server_key_iv: "5e8365f3ea36f56019bb639372d8c7c8"
server_key_version: "encryptionkey@0"
I20260806 02:59:32.891428 6377 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260806 02:59:32.895895 6392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:32.897823 6377 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260806 02:59:32.898063 6377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/master-1/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/master-1/wal
uuid: "001d2d57ebfd4d6aa7d81fde49a50231"
format_stamp: "Formatted at 2026-08-06 02:59:32 on dist-test-slave-dhph"
server_key: "f316790adaa744ecd098fc7f2e399fad"
server_key_iv: "5e8365f3ea36f56019bb639372d8c7c8"
server_key_version: "encryptionkey@0"
I20260806 02:59:32.898399 6377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:59:32.932488 6377 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:59:32.937893 6377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:59:32.938161 6377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:59:32.964038 6377 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.253:40869
I20260806 02:59:32.964057 6446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.253:40869 every 8 connection(s)
I20260806 02:59:32.965636 6377 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/master-1/data/info.pb
I20260806 02:59:32.969873 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 6377
I20260806 02:59:32.970306 6447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:59:32.981659 6447 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36051 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 02:59:32 dist-test-slave-dhph krb5kdc[5855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985169, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.74.253@KRBTEST.COM
Aug 06 02:59:32 dist-test-slave-dhph krb5kdc[5855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985172, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.74.253@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM
I20260806 02:59:33.000020 6450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:32.972593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48478 (local address 127.31.74.253:40869)
0806 02:59:32.973540 (+ 947us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:32.973553 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:32.973597 (+ 44us) server_negotiation.cc:407] Connection header received
0806 02:59:32.975700 (+ 2103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:32.975724 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:32.976174 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:32.976573 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:32.978716 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:32.980580 (+ 1864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:32.982823 (+ 2243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:32.983716 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:32.983873 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:32.989601 (+ 5728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:32.989610 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:32.989621 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:32.989667 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:32.995207 (+ 5540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:32.996247 (+ 1040us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:32.996256 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:32.996271 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:32.996490 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:32.997075 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:32.997079 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:32.997081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:32.997631 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:32.997977 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:32.998457 (+ 480us) server_negotiation.cc:299] Negotiation successful
0806 02:59:32.998981 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":724,"thread_start_us":440,"threads_started":1}
I20260806 02:59:33.006803 6452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:32.985224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46272 (local address 127.31.74.254:36051)
0806 02:59:32.986780 (+ 1556us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:32.986792 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:32.986809 (+ 17us) server_negotiation.cc:407] Connection header received
0806 02:59:32.986857 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:32.986861 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:32.986931 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:32.987025 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:32.988507 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:32.989632 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:32.992448 (+ 2816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:32.993032 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:32.993148 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:32.997610 (+ 4462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:32.997616 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:32.997619 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:32.997670 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:33.001286 (+ 3616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:33.003543 (+ 2257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:33.003549 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:33.003557 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:33.003737 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:33.004193 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:33.004208 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:33.004212 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:33.004653 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:33.004870 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:33.006046 (+ 1176us) server_negotiation.cc:299] Negotiation successful
0806 02:59:33.006312 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1447,"thread_start_us":1351,"threads_started":1}
I20260806 02:59:33.006863 6451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:32.985170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:36051 (local address 127.0.0.1:46272)
0806 02:59:32.986017 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:32.986082 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:32.986266 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:32.987142 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:32.987152 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:32.988028 (+ 876us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:59:32.988306 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:32.988316 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:32.989840 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:32.989852 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:32.992078 (+ 2226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:32.992105 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:32.993302 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:32.993312 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:32.993550 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:32.994230 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:59:32.994261 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:59:32.997380 (+ 3119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:59:33.001436 (+ 4056us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:33.001453 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:33.001461 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:33.003396 (+ 1935us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:33.003888 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:33.003894 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:33.003900 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:33.004052 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:33.004812 (+ 760us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:59:33.004823 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:59:33.005035 (+ 212us) client_negotiation.cc:769] Sending connection context
0806 02:59:33.005899 (+ 864us) client_negotiation.cc:240] Negotiation successful
0806 02:59:33.006315 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":362,"threads_started":1}
I20260806 02:59:33.009230 6447 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.74.253" port: 40869 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 02:59:33 dist-test-slave-dhph krb5kdc[5855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985172, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.74.253@KRBTEST.COM for oryx/127.31.74.253@KRBTEST.COM
I20260806 02:59:33.026759 6451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:33.010227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.253:40869 (local address 127.0.0.1:48488)
0806 02:59:33.010530 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:33.010550 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:33.010734 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:33.011401 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:33.011406 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:33.012039 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:59:33.012203 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:33.012216 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:33.013547 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:33.013552 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:33.014662 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:33.014688 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:33.015547 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:33.015552 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:33.015758 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:33.016484 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:59:33.016503 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:59:33.019232 (+ 2729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:59:33.023506 (+ 4274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:33.023511 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:33.023514 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:33.023959 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:33.024432 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:33.024436 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:33.024439 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:33.024598 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:33.025402 (+ 804us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:59:33.025408 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:59:33.025724 (+ 316us) client_negotiation.cc:769] Sending connection context
0806 02:59:33.025916 (+ 192us) client_negotiation.cc:240] Negotiation successful
0806 02:59:33.026285 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151}
I20260806 02:59:33.027350 6450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:33.010426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48488 (local address 127.31.74.253:40869)
0806 02:59:33.010676 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:33.010682 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:33.010701 (+ 19us) server_negotiation.cc:407] Connection header received
0806 02:59:33.011026 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:33.011030 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:33.011094 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:33.011200 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:33.012502 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:33.013339 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:33.014854 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:33.015368 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:33.015488 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:33.019485 (+ 3997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:33.019491 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:33.019494 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:33.019530 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:33.023316 (+ 3786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:33.024132 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:33.024137 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:33.024140 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:33.024271 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:33.024793 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:33.024797 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:33.024800 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:33.025147 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:33.025291 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:33.026145 (+ 854us) server_negotiation.cc:299] Negotiation successful
0806 02:59:33.026510 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":66}
I20260806 02:59:33.033982 6447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 001d2d57ebfd4d6aa7d81fde49a50231: Bootstrap starting.
I20260806 02:59:33.037881 6447 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 001d2d57ebfd4d6aa7d81fde49a50231: Neither blocks nor log segments found. Creating new log.
I20260806 02:59:33.039320 6447 log.cc:824] T 00000000000000000000000000000000 P 001d2d57ebfd4d6aa7d81fde49a50231: Log is configured to *not* fsync() on all Append() calls
I20260806 02:59:33.042817 6447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 001d2d57ebfd4d6aa7d81fde49a50231: No bootstrap required, opened a new log
I20260806 02:59:33.052348 6447 raft_consensus.cc:348] T 00000000000000000000000000000000 P 001d2d57ebfd4d6aa7d81fde49a50231 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec7ce3806c9d48a2bbbfacaed8423300" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36051 } } peers { permanent_uuid: "001d2d57ebfd4d6aa7d81fde49a50231" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 40869 } }
I20260806 02:59:33.052798 6447 raft_consensus.cc:374] T 00000000000000000000000000000000 P 001d2d57ebfd4d6aa7d81fde49a50231 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:59:33.052876 6447 raft_consensus.cc:729] T 00000000000000000000000000000000 P 001d2d57ebfd4d6aa7d81fde49a50231 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 001d2d57ebfd4d6aa7d81fde49a50231, State: Initialized, Role: FOLLOWER
I20260806 02:59:33.053685 6447 consensus_queue.cc:260] T 00000000000000000000000000000000 P 001d2d57ebfd4d6aa7d81fde49a50231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec7ce3806c9d48a2bbbfacaed8423300" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36051 } } peers { permanent_uuid: "001d2d57ebfd4d6aa7d81fde49a50231" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 40869 } }
I20260806 02:59:33.054922 6454 sys_catalog.cc:455] T 00000000000000000000000000000000 P 001d2d57ebfd4d6aa7d81fde49a50231 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec7ce3806c9d48a2bbbfacaed8423300" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36051 } } peers { permanent_uuid: "001d2d57ebfd4d6aa7d81fde49a50231" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 40869 } } }
I20260806 02:59:33.055351 6454 sys_catalog.cc:458] T 00000000000000000000000000000000 P 001d2d57ebfd4d6aa7d81fde49a50231 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:59:33.056368 6447 sys_catalog.cc:564] T 00000000000000000000000000000000 P 001d2d57ebfd4d6aa7d81fde49a50231 [sys.catalog]: configured and running, proceeding with master startup.
W20260806 02:59:33.067313 6465 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 001d2d57ebfd4d6aa7d81fde49a50231: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:59:33.067504 6465 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:59:33.070879 6377 master_runner.cc:428] Detected that this master 001d2d57ebfd4d6aa7d81fde49a50231 is joining an existing cluster
I20260806 02:59:33.071002 6377 master_runner.cc:432] Initiating AddMaster RPC to add 127.31.74.253:40869
I20260806 02:59:33.096094 6450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:33.076160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48496 (local address 127.31.74.253:40869)
0806 02:59:33.076387 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:33.076396 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:33.079671 (+ 3275us) server_negotiation.cc:407] Connection header received
0806 02:59:33.079811 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:33.079816 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:33.079875 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:33.079969 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:33.080954 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:33.081720 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:33.083193 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:33.083896 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:33.084039 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:33.087278 (+ 3239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:33.087287 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:33.087295 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:33.087333 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:33.092029 (+ 4696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:33.093066 (+ 1037us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:33.093071 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:33.093073 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:33.093462 (+ 389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:33.094089 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:33.094095 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:33.094099 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:33.094532 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:33.094763 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:33.095213 (+ 450us) server_negotiation.cc:299] Negotiation successful
0806 02:59:33.095654 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":148,"server-negotiator.queue_time_us":45,"spinlock_wait_cycles":28928}
I20260806 02:59:33.096133 6472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:33.076143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.253:40869 (local address 127.0.0.1:48496)
0806 02:59:33.079573 (+ 3430us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:33.079591 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:33.079706 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:33.079987 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:33.079991 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:33.080559 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:59:33.080813 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:33.080822 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:33.081907 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:33.081912 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:33.083051 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:33.083062 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:33.084124 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:33.084132 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:33.084379 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:33.085369 (+ 990us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:59:33.085399 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:59:33.086981 (+ 1582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:59:33.092222 (+ 5241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:33.092229 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:33.092234 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:33.092886 (+ 652us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:33.093693 (+ 807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:33.093700 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:33.093705 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:33.093940 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:33.094696 (+ 756us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:59:33.094701 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:59:33.094872 (+ 171us) client_negotiation.cc:769] Sending connection context
0806 02:59:33.095053 (+ 181us) client_negotiation.cc:240] Negotiation successful
0806 02:59:33.095461 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3264,"mutex_wait_us":3,"spinlock_wait_cycles":1408,"thread_start_us":2983,"threads_started":1}
I20260806 02:59:33.096556 6452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:33.076725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46276 (local address 127.31.74.254:36051)
0806 02:59:33.076922 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:33.076930 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:33.077063 (+ 133us) server_negotiation.cc:407] Connection header received
0806 02:59:33.077412 (+ 349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:33.077418 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:33.077520 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:33.077642 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:33.079597 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:33.080824 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:33.083318 (+ 2494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:33.083973 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:33.084103 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:33.087124 (+ 3021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:33.087130 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:33.087140 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:33.087215 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:33.092029 (+ 4814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:33.093934 (+ 1905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:33.093941 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:33.093946 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:33.094150 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:33.094608 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:33.094617 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:33.094624 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:33.095037 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:33.095217 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:33.095895 (+ 678us) server_negotiation.cc:299] Negotiation successful
0806 02:59:33.096202 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260806 02:59:33.098374 6471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:33.076062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:36051 (local address 127.0.0.1:46276)
0806 02:59:33.076935 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:33.076959 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:33.077193 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:33.077677 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:33.077682 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:33.078281 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:59:33.079319 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:33.079342 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:33.081071 (+ 1729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:33.081077 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:33.083126 (+ 2049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:33.083149 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:33.084352 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:33.084357 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:33.084486 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:33.085480 (+ 994us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:59:33.085500 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:59:33.086876 (+ 1376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:59:33.093052 (+ 6176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:33.093061 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:33.093066 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:33.093757 (+ 691us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:33.094280 (+ 523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:33.094285 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:33.094290 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:33.094442 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:33.095291 (+ 849us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:59:33.095297 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:59:33.095536 (+ 239us) client_negotiation.cc:769] Sending connection context
0806 02:59:33.096842 (+ 1306us) client_negotiation.cc:240] Negotiation successful
0806 02:59:33.097995 (+ 1153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":704,"mutex_wait_us":3,"thread_start_us":845,"threads_started":1}
Aug 06 02:59:33 dist-test-slave-dhph krb5kdc[5855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985170, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.74.254@KRBTEST.COM for oryx/127.31.74.253@KRBTEST.COM
I20260806 02:59:33.132759 6450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:33.107981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48498 (local address 127.31.74.253:40869)
0806 02:59:33.108569 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:33.108577 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:33.111463 (+ 2886us) server_negotiation.cc:407] Connection header received
0806 02:59:33.112005 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:33.112013 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:33.112145 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:33.112280 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:33.114203 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:33.115524 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:33.117920 (+ 2396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:33.118828 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:33.118986 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:33.123877 (+ 4891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:33.123888 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:33.123901 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:33.123958 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:33.128296 (+ 4338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:33.129468 (+ 1172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:33.129480 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:33.129485 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:33.129759 (+ 274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:33.130287 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:33.130292 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:33.130296 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:33.130791 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:33.130944 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:33.132013 (+ 1069us) server_negotiation.cc:299] Negotiation successful
0806 02:59:33.132344 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264}
I20260806 02:59:33.132828 6473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:33.108001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.253:40869 (local address 127.0.0.1:48498)
0806 02:59:33.111275 (+ 3274us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:33.111335 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:33.111669 (+ 334us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:33.112368 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:33.112377 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:33.113563 (+ 1186us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:59:33.113960 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:33.113977 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:33.115733 (+ 1756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:33.115740 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:33.117710 (+ 1970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:33.117736 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:33.119050 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:33.119058 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:33.119307 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:33.120230 (+ 923us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:59:33.120260 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:59:33.123571 (+ 3311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:59:33.128510 (+ 4939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:33.128533 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:33.128544 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:33.129309 (+ 765us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:33.129928 (+ 619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:33.129936 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:33.129943 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:33.130155 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:33.130972 (+ 817us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:59:33.130984 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:59:33.131391 (+ 407us) client_negotiation.cc:769] Sending connection context
0806 02:59:33.131688 (+ 297us) client_negotiation.cc:240] Negotiation successful
0806 02:59:33.132177 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2974,"thread_start_us":2780,"threads_started":1}
I20260806 02:59:33.135780 5904 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.31.74.253:40869: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "001d2d57ebfd4d6aa7d81fde49a50231"
member_type: NON_VOTER
last_known_addr {
host: "127.31.74.253"
port: 40869
}
attrs {
promote: true
}
}
dest_uuid: "ec7ce3806c9d48a2bbbfacaed8423300"
cas_config_opid_index: -1
I20260806 02:59:33.137259 5904 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec7ce3806c9d48a2bbbfacaed8423300 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ec7ce3806c9d48a2bbbfacaed8423300" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36051 } } peers { permanent_uuid: "001d2d57ebfd4d6aa7d81fde49a50231" member_type: NON_VOTER last_known_addr { host: "127.31.74.253" port: 40869 } attrs { promote: true } }
WARNING: no policy specified for oryx/127.31.74.252@KRBTEST.COM; defaulting to no policy
I20260806 02:59:33.141531 6477 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ec7ce3806c9d48a2bbbfacaed8423300 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 001d2d57ebfd4d6aa7d81fde49a50231 (127.31.74.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ec7ce3806c9d48a2bbbfacaed8423300" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36051 } } peers { permanent_uuid: "001d2d57ebfd4d6aa7d81fde49a50231" member_type: NON_VOTER last_known_addr { host: "127.31.74.253" port: 40869 } attrs { promote: true } } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.74.252@KRBTEST.COM" created.
I20260806 02:59:33.142908 6420 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 001d2d57ebfd4d6aa7d81fde49a50231 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:59:33.146540 6477 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.31.74.253:40869
I20260806 02:59:33.147449 6476 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec7ce3806c9d48a2bbbfacaed8423300 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ec7ce3806c9d48a2bbbfacaed8423300" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ec7ce3806c9d48a2bbbfacaed8423300" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36051 } } peers { permanent_uuid: "001d2d57ebfd4d6aa7d81fde49a50231" member_type: NON_VOTER last_known_addr { host: "127.31.74.253" port: 40869 } attrs { promote: true } } }
I20260806 02:59:33.147806 6377 master.cc:537] Master@127.31.74.253:40869 shutting down...
I20260806 02:59:33.148011 6476 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec7ce3806c9d48a2bbbfacaed8423300 [sys.catalog]: This master's current role is: LEADER
I20260806 02:59:33.158653 6420 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 001d2d57ebfd4d6aa7d81fde49a50231 [term 1 FOLLOWER]: Refusing update from remote peer ec7ce3806c9d48a2bbbfacaed8423300: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260806 02:59:33.161770 6476 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ec7ce3806c9d48a2bbbfacaed8423300 [LEADER]: Connected to new peer: Peer: permanent_uuid: "001d2d57ebfd4d6aa7d81fde49a50231" member_type: NON_VOTER last_known_addr { host: "127.31.74.253" port: 40869 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:59:33.167589 6377 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 001d2d57ebfd4d6aa7d81fde49a50231 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:59:33.168118 6377 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 001d2d57ebfd4d6aa7d81fde49a50231 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:59:33.168355 6377 tablet_replica.cc:330] T 00000000000000000000000000000000 P 001d2d57ebfd4d6aa7d81fde49a50231: stopping tablet replica
W20260806 02:59:33.183578 5888 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260806 02:59:33.194352 6377 master.cc:552] Master@127.31.74.253:40869 shutdown complete.
I20260806 02:59:33.194546 6377 master_runner.cc:305] Clearing existing system tablet
I20260806 02:59:33.195025 6377 fs_manager.cc:359] Metadata directory not provided
I20260806 02:59:33.195155 6377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:59:33.198475 6377 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.74.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/krb5kdc/oryx_127.31.74.252.keytab.
Entry for principal oryx/127.31.74.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/krb5kdc/oryx_127.31.74.252.keytab.
I20260806 02:59:33.201215 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.252:35959
--webserver_interface=127.31.74.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/krb5kdc/oryx_127.31.74.252.keytab
--principal=oryx/127.31.74.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:40361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.74.254:36051,127.31.74.253:40869,127.31.74.252:35959
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:59:33.204543 6482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:33.204713 6473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:33.203933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.253:40869 (local address 127.0.0.1:48500)
0806 02:59:33.204334 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:33.204494 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.253:40869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260806 02:59:33.205104 5888 consensus_peers.cc:487] T 00000000000000000000000000000000 P ec7ce3806c9d48a2bbbfacaed8423300 -> Peer 001d2d57ebfd4d6aa7d81fde49a50231 (127.31.74.253:40869): Couldn't send request to peer 001d2d57ebfd4d6aa7d81fde49a50231. Status: Network error: Client connection negotiation failed: client connection to 127.31.74.253:40869: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260806 02:59:33.205055 6377 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:59:33.205550 6377 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.006s sys 0.000s
I20260806 02:59:33.205677 6377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/master-1/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/master-1/wal
uuid: "001d2d57ebfd4d6aa7d81fde49a50231"
format_stamp: "Formatted at 2026-08-06 02:59:32 on dist-test-slave-dhph"
server_key: "f316790adaa744ecd098fc7f2e399fad"
server_key_iv: "5e8365f3ea36f56019bb639372d8c7c8"
server_key_version: "encryptionkey@0"
I20260806 02:59:33.210033 6377 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 001d2d57ebfd4d6aa7d81fde49a50231: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:59:33.216245 6377 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 001d2d57ebfd4d6aa7d81fde49a50231: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260806 02:59:33.216524 6377 log.cc:1196] T 00000000000000000000000000000000 P 001d2d57ebfd4d6aa7d81fde49a50231: Deleting WAL directory at /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260806 02:59:33.217005 6377 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 001d2d57ebfd4d6aa7d81fde49a50231: Deleting consensus metadata
I20260806 02:59:33.217338 6377 master_runner.cc:315] Copying system tablet from 127.31.74.254:36051
I20260806 02:59:33.220460 6377 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 001d2d57ebfd4d6aa7d81fde49a50231: tablet copy: Beginning tablet copy session from remote peer at address 127.31.74.254:36051
I20260806 02:59:33.247526 6452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:33.221892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46290 (local address 127.31.74.254:36051)
0806 02:59:33.222098 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:33.222106 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:33.229480 (+ 7374us) server_negotiation.cc:407] Connection header received
0806 02:59:33.229768 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:33.229775 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:33.229880 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:33.230015 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:33.231468 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:33.232766 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:33.235466 (+ 2700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:33.236161 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:33.236344 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:33.239408 (+ 3064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:33.239415 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:33.239420 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:33.239463 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:33.243606 (+ 4143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:33.244630 (+ 1024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:33.244637 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:33.244641 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:33.244818 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:33.245377 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:33.245383 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:33.245388 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:33.245832 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:33.246017 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:33.246824 (+ 807us) server_negotiation.cc:299] Negotiation successful
0806 02:59:33.247080 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 02:59:33.247500 6489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:33.221510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:36051 (local address 127.0.0.1:46290)
0806 02:59:33.229273 (+ 7763us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:33.229310 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:33.229546 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:33.230088 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:33.230096 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:33.230828 (+ 732us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:59:33.231257 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:33.231277 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:33.232980 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:33.232989 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:33.235183 (+ 2194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:33.235219 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:33.236381 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:33.236392 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:33.236677 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:33.237578 (+ 901us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:59:33.237613 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:59:33.239101 (+ 1488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:59:33.243776 (+ 4675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:33.243784 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:33.243789 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:33.244463 (+ 674us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:33.245007 (+ 544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:33.245013 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:33.245017 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:33.245219 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:33.246088 (+ 869us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:59:33.246093 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:59:33.246298 (+ 205us) client_negotiation.cc:769] Sending connection context
0806 02:59:33.246505 (+ 207us) client_negotiation.cc:240] Negotiation successful
0806 02:59:33.246941 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7521,"thread_start_us":7332,"threads_started":1}
I20260806 02:59:33.248615 5924 tablet_copy_service.cc:140] P ec7ce3806c9d48a2bbbfacaed8423300: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 001d2d57ebfd4d6aa7d81fde49a50231 ({username='oryx', principal='oryx/127.31.74.253@KRBTEST.COM'} at 127.0.0.1:46290)
I20260806 02:59:33.248937 5924 tablet_copy_service.cc:161] P ec7ce3806c9d48a2bbbfacaed8423300: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 001d2d57ebfd4d6aa7d81fde49a50231 at {username='oryx', principal='oryx/127.31.74.253@KRBTEST.COM'} at 127.0.0.1:46290: session id = 001d2d57ebfd4d6aa7d81fde49a50231-00000000000000000000000000000000
I20260806 02:59:33.254079 5924 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P ec7ce3806c9d48a2bbbfacaed8423300: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 02:59:33.256994 6377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260806 02:59:33.265095 6377 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 001d2d57ebfd4d6aa7d81fde49a50231: tablet copy: Starting download of 0 data blocks...
I20260806 02:59:33.265501 6377 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 001d2d57ebfd4d6aa7d81fde49a50231: tablet copy: Starting download of 1 WAL segments...
I20260806 02:59:33.269711 6377 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 001d2d57ebfd4d6aa7d81fde49a50231: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 02:59:33.276023 5924 tablet_copy_service.cc:342] P ec7ce3806c9d48a2bbbfacaed8423300: Request end of tablet copy session 001d2d57ebfd4d6aa7d81fde49a50231-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.31.74.253@KRBTEST.COM'} at 127.0.0.1:46290
I20260806 02:59:33.276356 5924 tablet_copy_service.cc:433] P ec7ce3806c9d48a2bbbfacaed8423300: ending tablet copy session 001d2d57ebfd4d6aa7d81fde49a50231-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 001d2d57ebfd4d6aa7d81fde49a50231
I20260806 02:59:33.297695 6377 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:59:33.301191 6490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:33.312156 6494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:59:33.313443 6371 heartbeater.cc:503] Master 127.31.74.254:36051 was elected leader, sending a full tablet report...
I20260806 02:59:33.315618 6377 server_base.cc:1034] running on GCE node
W20260806 02:59:33.315997 6491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:59:33.316565 6377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:59:33.330922 6377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:59:33.332455 6377 hybrid_clock.cc:648] HybridClock initialized: now 1785985173332406 us; error 48 us; skew 500 ppm
I20260806 02:59:33.332791 6377 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:59:33 dist-test-slave-dhph krb5kdc[5855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985173, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.74.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:59:33.335053 6377 init.cc:377] Logged in from keytab as oryx/127.31.74.253@KRBTEST.COM (short username oryx)
I20260806 02:59:33.339820 6377 webserver.cc:460] Webserver started at http://127.31.74.253:35789/ using document root <none> and password file <none>
I20260806 02:59:33.340217 6377 fs_manager.cc:359] Metadata directory not provided
I20260806 02:59:33.340319 6377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:59:33.343304 6377 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:59:33.345552 6500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:33.346035 6377 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 02:59:33.346218 6377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/master-1/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/master-1/wal
uuid: "001d2d57ebfd4d6aa7d81fde49a50231"
format_stamp: "Formatted at 2026-08-06 02:59:32 on dist-test-slave-dhph"
server_key: "f316790adaa744ecd098fc7f2e399fad"
server_key_iv: "5e8365f3ea36f56019bb639372d8c7c8"
server_key_version: "encryptionkey@0"
I20260806 02:59:33.346414 6377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:59:33.362829 6377 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:59:33.368284 6377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:59:33.408321 6377 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.253:40869
I20260806 02:59:33.408356 6552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.253:40869 every 8 connection(s)
I20260806 02:59:33.409246 6377 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/master-1/data/info.pb
I20260806 02:59:33.416818 6553 sys_catalog.cc:263] Verifying existing consensus state
I20260806 02:59:33.421677 6553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 001d2d57ebfd4d6aa7d81fde49a50231: Bootstrap starting.
I20260806 02:59:33.443418 6553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 001d2d57ebfd4d6aa7d81fde49a50231: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 02:59:33.444028 6553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 001d2d57ebfd4d6aa7d81fde49a50231: Bootstrap complete.
I20260806 02:59:33.445611 6553 raft_consensus.cc:348] T 00000000000000000000000000000000 P 001d2d57ebfd4d6aa7d81fde49a50231 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ec7ce3806c9d48a2bbbfacaed8423300" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36051 } } peers { permanent_uuid: "001d2d57ebfd4d6aa7d81fde49a50231" member_type: NON_VOTER last_known_addr { host: "127.31.74.253" port: 40869 } attrs { promote: true } }
I20260806 02:59:33.445986 6553 raft_consensus.cc:729] T 00000000000000000000000000000000 P 001d2d57ebfd4d6aa7d81fde49a50231 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 001d2d57ebfd4d6aa7d81fde49a50231, State: Initialized, Role: LEARNER
I20260806 02:59:33.446316 6553 consensus_queue.cc:260] T 00000000000000000000000000000000 P 001d2d57ebfd4d6aa7d81fde49a50231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ec7ce3806c9d48a2bbbfacaed8423300" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36051 } } peers { permanent_uuid: "001d2d57ebfd4d6aa7d81fde49a50231" member_type: NON_VOTER last_known_addr { host: "127.31.74.253" port: 40869 } attrs { promote: true } }
I20260806 02:59:33.447412 6556 sys_catalog.cc:455] T 00000000000000000000000000000000 P 001d2d57ebfd4d6aa7d81fde49a50231 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ec7ce3806c9d48a2bbbfacaed8423300" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36051 } } peers { permanent_uuid: "001d2d57ebfd4d6aa7d81fde49a50231" member_type: NON_VOTER last_known_addr { host: "127.31.74.253" port: 40869 } attrs { promote: true } } }
I20260806 02:59:33.447723 6553 sys_catalog.cc:564] T 00000000000000000000000000000000 P 001d2d57ebfd4d6aa7d81fde49a50231 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:59:33.447964 6556 sys_catalog.cc:458] T 00000000000000000000000000000000 P 001d2d57ebfd4d6aa7d81fde49a50231 [sys.catalog]: This master's current role is: LEARNER
I20260806 02:59:33.469913 6567 catalog_manager.cc:1259] Loaded cluster ID: 8e6a631af4aa433eb774f4779a94b441
I20260806 02:59:33.470068 6567 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 001d2d57ebfd4d6aa7d81fde49a50231: loading cluster ID for follower catalog manager: success
I20260806 02:59:33.474185 6567 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 001d2d57ebfd4d6aa7d81fde49a50231: acquiring CA information for follower catalog manager: success
I20260806 02:59:33.476966 6567 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 001d2d57ebfd4d6aa7d81fde49a50231: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 02:59:33.589577 6473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:33.579851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.253:40869 (local address 127.0.0.1:48512)
0806 02:59:33.580131 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:33.580153 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:33.580281 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:33.581478 (+ 1197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:33.581485 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:33.581527 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:59:33.581759 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:33.581772 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:33.583839 (+ 2067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:33.583847 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:33.586320 (+ 2473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:33.586359 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:33.588458 (+ 2099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:33.588467 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:33.588764 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:33.588847 (+ 83us) client_negotiation.cc:769] Sending connection context
0806 02:59:33.588980 (+ 133us) client_negotiation.cc:240] Negotiation successful
0806 02:59:33.589189 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154}
I20260806 02:59:33.589633 6568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:33.580157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48512 (local address 127.31.74.253:40869)
0806 02:59:33.581002 (+ 845us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:33.581009 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:33.581034 (+ 25us) server_negotiation.cc:407] Connection header received
0806 02:59:33.581120 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:33.581126 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:33.581232 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:33.581387 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:59:33.581935 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:33.583627 (+ 1692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:33.586608 (+ 2981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:33.588180 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:33.588359 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:33.588637 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:33.589030 (+ 393us) server_negotiation.cc:299] Negotiation successful
0806 02:59:33.589201 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":688,"thread_start_us":376,"threads_started":1}
I20260806 02:59:33.591533 6527 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 001d2d57ebfd4d6aa7d81fde49a50231 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260806 02:59:33.592264 6556 sys_catalog.cc:455] T 00000000000000000000000000000000 P 001d2d57ebfd4d6aa7d81fde49a50231 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec7ce3806c9d48a2bbbfacaed8423300. Latest consensus state: current_term: 1 leader_uuid: "ec7ce3806c9d48a2bbbfacaed8423300" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ec7ce3806c9d48a2bbbfacaed8423300" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36051 } } peers { permanent_uuid: "001d2d57ebfd4d6aa7d81fde49a50231" member_type: NON_VOTER last_known_addr { host: "127.31.74.253" port: 40869 } attrs { promote: true } } }
I20260806 02:59:33.592602 6556 sys_catalog.cc:458] T 00000000000000000000000000000000 P 001d2d57ebfd4d6aa7d81fde49a50231 [sys.catalog]: This master's current role is: LEARNER
W20260806 02:59:33.672979 6483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:59:33.673345 6483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:59:33.673518 6483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:59:33.681149 6483 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:59:33.681313 6483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:59:33.681439 6483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:59:33.681543 6483 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:59:33.681629 6483 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:59:33.692191 6483 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:40361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.31.74.254:36051,127.31.74.253:40869,127.31.74.252:35959
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/krb5kdc/oryx_127.31.74.252.keytab
--principal=oryx/127.31.74.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.252:35959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.31.74.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:59:33.693663 6483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:59:33.695034 6483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:59:33.708071 6574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:33.708159 6573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:33.708398 6576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:59:33.709486 6483 server_base.cc:1034] running on GCE node
I20260806 02:59:33.710464 6483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:59:33.712134 6483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:59:33.713503 6483 hybrid_clock.cc:648] HybridClock initialized: now 1785985173713444 us; error 44 us; skew 500 ppm
I20260806 02:59:33.713963 6483 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:59:33 dist-test-slave-dhph krb5kdc[5855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985173, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.74.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:59:33.717912 6483 init.cc:377] Logged in from keytab as oryx/127.31.74.252@KRBTEST.COM (short username oryx)
I20260806 02:59:33.719794 6483 webserver.cc:460] Webserver started at http://127.31.74.252:45045/ using document root <none> and password file <none>
I20260806 02:59:33.720647 6483 fs_manager.cc:359] Metadata directory not provided
I20260806 02:59:33.720813 6483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:59:33.721129 6483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:59:33.724418 6483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/master-2/data/instance:
uuid: "2698ead82bb0481bbc3bfee291d34fdc"
format_stamp: "Formatted at 2026-08-06 02:59:33 on dist-test-slave-dhph"
server_key: "29426f20039b38a4f09c6c85eb1b87c8"
server_key_iv: "a22eefb151896bbaa5fb98d0c0cac467"
server_key_version: "encryptionkey@0"
I20260806 02:59:33.725246 6483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/master-2/wal/instance:
uuid: "2698ead82bb0481bbc3bfee291d34fdc"
format_stamp: "Formatted at 2026-08-06 02:59:33 on dist-test-slave-dhph"
server_key: "29426f20039b38a4f09c6c85eb1b87c8"
server_key_iv: "a22eefb151896bbaa5fb98d0c0cac467"
server_key_version: "encryptionkey@0"
I20260806 02:59:33.731359 6483 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260806 02:59:33.735467 6583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:33.737069 6483 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:59:33.737284 6483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/master-2/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/master-2/wal
uuid: "2698ead82bb0481bbc3bfee291d34fdc"
format_stamp: "Formatted at 2026-08-06 02:59:33 on dist-test-slave-dhph"
server_key: "29426f20039b38a4f09c6c85eb1b87c8"
server_key_iv: "a22eefb151896bbaa5fb98d0c0cac467"
server_key_version: "encryptionkey@0"
I20260806 02:59:33.737506 6483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:59:33.752508 6483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:59:33.757706 6483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:59:33.758072 6483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:59:33.781014 6483 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.252:35959
I20260806 02:59:33.781117 6636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.252:35959 every 8 connection(s)
I20260806 02:59:33.782574 6483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/master-2/data/info.pb
I20260806 02:59:33.786965 6637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:59:33.791172 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 6483
I20260806 02:59:33.794719 6637 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36051 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 02:59:33 dist-test-slave-dhph krb5kdc[5855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985173, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.74.252@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM
Aug 06 02:59:33 dist-test-slave-dhph krb5kdc[5855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985169, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.74.252@KRBTEST.COM
I20260806 02:59:33.817418 6642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:33.799200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46296 (local address 127.31.74.254:36051)
0806 02:59:33.800341 (+ 1141us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:33.800346 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:33.800362 (+ 16us) server_negotiation.cc:407] Connection header received
0806 02:59:33.800412 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:33.800416 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:33.800474 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:33.800564 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:33.802220 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:33.803361 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:33.805776 (+ 2415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:33.806366 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:33.806534 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:33.810231 (+ 3697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:33.810240 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:33.810247 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:33.810328 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:33.813431 (+ 3103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:33.814474 (+ 1043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:33.814478 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:33.814480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:33.814678 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:33.815232 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:33.815239 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:33.815242 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:33.815694 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:33.815868 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:33.816850 (+ 982us) server_negotiation.cc:299] Negotiation successful
0806 02:59:33.817055 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1039,"thread_start_us":922,"threads_started":1}
I20260806 02:59:33.819243 6640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:33.793495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39596 (local address 127.31.74.252:35959)
0806 02:59:33.794339 (+ 844us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:33.794352 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:33.794394 (+ 42us) server_negotiation.cc:407] Connection header received
0806 02:59:33.797057 (+ 2663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:33.797082 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:33.797551 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:33.797966 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:33.800219 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:33.802268 (+ 2049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:33.803941 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:33.805112 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:33.805530 (+ 418us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:33.811245 (+ 5715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:33.811260 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:33.811280 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:33.811328 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:33.815036 (+ 3708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:33.816081 (+ 1045us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:33.816090 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:33.816094 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:33.816275 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:33.816800 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:33.816804 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:33.816808 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:33.817270 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:33.817620 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:33.818176 (+ 556us) server_negotiation.cc:299] Negotiation successful
0806 02:59:33.818440 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":618,"thread_start_us":374,"threads_started":1}
I20260806 02:59:33.819257 6641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:33.798913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:36051 (local address 127.0.0.1:46296)
0806 02:59:33.799691 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:33.799745 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:33.799904 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:33.800672 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:33.800680 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:33.801596 (+ 916us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:59:33.802019 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:33.802038 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:33.803531 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:33.803535 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:33.805605 (+ 2070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:33.805621 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:33.806703 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:33.806710 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:33.806856 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:33.807461 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:59:33.807482 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:59:33.809995 (+ 2513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:59:33.813575 (+ 3580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:33.813589 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:33.813608 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:33.814316 (+ 708us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:33.814794 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:33.814800 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:33.814805 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:33.815057 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:33.815865 (+ 808us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:59:33.815872 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:59:33.816415 (+ 543us) client_negotiation.cc:769] Sending connection context
0806 02:59:33.816698 (+ 283us) client_negotiation.cc:240] Negotiation successful
0806 02:59:33.817335 (+ 637us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"mutex_wait_us":1,"thread_start_us":323,"threads_started":1}
I20260806 02:59:33.822190 6637 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.74.253" port: 40869 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 02:59:33 dist-test-slave-dhph krb5kdc[5855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985173, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.74.252@KRBTEST.COM for oryx/127.31.74.253@KRBTEST.COM
I20260806 02:59:33.839097 6641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:33.823172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.253:40869 (local address 127.0.0.1:48516)
0806 02:59:33.823393 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:33.823415 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:33.823519 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:33.823861 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:33.823864 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:33.824289 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:59:33.824451 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:33.824460 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:33.825713 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:33.825717 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:33.826940 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:33.826953 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:33.827847 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:33.827852 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:33.828060 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:33.828801 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:59:33.828821 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:59:33.831479 (+ 2658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:59:33.835955 (+ 4476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:33.835960 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:33.835963 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:33.836540 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:33.837113 (+ 573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:33.837118 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:33.837122 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:33.837274 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:33.837968 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:59:33.837973 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:59:33.838232 (+ 259us) client_negotiation.cc:769] Sending connection context
0806 02:59:33.838389 (+ 157us) client_negotiation.cc:240] Negotiation successful
0806 02:59:33.838740 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260806 02:59:33.839169 6568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:33.823323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48516 (local address 127.31.74.253:40869)
0806 02:59:33.823515 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:33.823519 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:33.823536 (+ 17us) server_negotiation.cc:407] Connection header received
0806 02:59:33.823610 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:33.823613 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:33.823674 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:33.823762 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:33.824646 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:33.825539 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:33.827095 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:33.827542 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:33.827753 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:33.831734 (+ 3981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:33.831739 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:33.831742 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:33.831779 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:33.835705 (+ 3926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:33.836744 (+ 1039us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:33.836749 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:33.836752 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:33.836944 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:33.837406 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:33.837411 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:33.837413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:33.837800 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:33.837945 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:33.838557 (+ 612us) server_negotiation.cc:299] Negotiation successful
0806 02:59:33.838887 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260806 02:59:33.840508 6637 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.74.252" port: 35959 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 02:59:33 dist-test-slave-dhph krb5kdc[5855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985173, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.74.252@KRBTEST.COM for oryx/127.31.74.252@KRBTEST.COM
I20260806 02:59:33.857362 6641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:33.841311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.252:35959 (local address 127.0.0.1:39610)
0806 02:59:33.841545 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:33.841563 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:33.841705 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:33.842096 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:33.842101 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:33.842684 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:59:33.842836 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:33.842844 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:33.843864 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:33.843871 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:33.844770 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:33.844781 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:33.845535 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:33.845540 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:33.845760 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:33.846566 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:59:33.846621 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:59:33.850669 (+ 4048us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:59:33.854343 (+ 3674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:33.854352 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:33.854357 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:33.854979 (+ 622us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:33.855406 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:33.855414 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:33.855419 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:33.855588 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:33.856227 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:59:33.856232 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:59:33.856382 (+ 150us) client_negotiation.cc:769] Sending connection context
0806 02:59:33.856542 (+ 160us) client_negotiation.cc:240] Negotiation successful
0806 02:59:33.856865 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84}
I20260806 02:59:33.857398 6640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:33.841584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39610 (local address 127.31.74.252:35959)
0806 02:59:33.841814 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:33.841822 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:33.841840 (+ 18us) server_negotiation.cc:407] Connection header received
0806 02:59:33.841889 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:33.841892 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:33.841959 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:33.842101 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:33.842980 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:33.843679 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:33.844915 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:33.845381 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:33.845519 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:33.850956 (+ 5437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:33.850961 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:33.850965 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:33.850999 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:33.854151 (+ 3152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:33.855113 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:33.855131 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:33.855141 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:33.855280 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:33.855753 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:33.855757 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:33.855759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:33.856081 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:33.856269 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:33.856640 (+ 371us) server_negotiation.cc:299] Negotiation successful
0806 02:59:33.857013 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":152,"server-negotiator.queue_time_us":95}
I20260806 02:59:33.863459 6637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2698ead82bb0481bbc3bfee291d34fdc: Bootstrap starting.
I20260806 02:59:33.867225 6637 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2698ead82bb0481bbc3bfee291d34fdc: Neither blocks nor log segments found. Creating new log.
I20260806 02:59:33.869159 6637 log.cc:824] T 00000000000000000000000000000000 P 2698ead82bb0481bbc3bfee291d34fdc: Log is configured to *not* fsync() on all Append() calls
I20260806 02:59:33.872581 6637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2698ead82bb0481bbc3bfee291d34fdc: No bootstrap required, opened a new log
I20260806 02:59:33.881341 6637 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2698ead82bb0481bbc3bfee291d34fdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec7ce3806c9d48a2bbbfacaed8423300" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36051 } } peers { permanent_uuid: "001d2d57ebfd4d6aa7d81fde49a50231" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 40869 } } peers { permanent_uuid: "2698ead82bb0481bbc3bfee291d34fdc" member_type: VOTER last_known_addr { host: "127.31.74.252" port: 35959 } }
I20260806 02:59:33.881722 6637 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2698ead82bb0481bbc3bfee291d34fdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:59:33.881848 6637 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2698ead82bb0481bbc3bfee291d34fdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2698ead82bb0481bbc3bfee291d34fdc, State: Initialized, Role: FOLLOWER
I20260806 02:59:33.882470 6637 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2698ead82bb0481bbc3bfee291d34fdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec7ce3806c9d48a2bbbfacaed8423300" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36051 } } peers { permanent_uuid: "001d2d57ebfd4d6aa7d81fde49a50231" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 40869 } } peers { permanent_uuid: "2698ead82bb0481bbc3bfee291d34fdc" member_type: VOTER last_known_addr { host: "127.31.74.252" port: 35959 } }
I20260806 02:59:33.883541 6644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2698ead82bb0481bbc3bfee291d34fdc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec7ce3806c9d48a2bbbfacaed8423300" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36051 } } peers { permanent_uuid: "001d2d57ebfd4d6aa7d81fde49a50231" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 40869 } } peers { permanent_uuid: "2698ead82bb0481bbc3bfee291d34fdc" member_type: VOTER last_known_addr { host: "127.31.74.252" port: 35959 } } }
I20260806 02:59:33.884020 6644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2698ead82bb0481bbc3bfee291d34fdc [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:59:33.884593 6637 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2698ead82bb0481bbc3bfee291d34fdc [sys.catalog]: configured and running, proceeding with master startup.
W20260806 02:59:33.895998 6656 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2698ead82bb0481bbc3bfee291d34fdc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:59:33.896315 6656 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:59:33.900867 6483 master_runner.cc:428] Detected that this master 2698ead82bb0481bbc3bfee291d34fdc is joining an existing cluster
I20260806 02:59:33.901038 6483 master_runner.cc:432] Initiating AddMaster RPC to add 127.31.74.252:35959
I20260806 02:59:33.926945 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 5965
I20260806 02:59:33.932701 6642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:33.906371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46310 (local address 127.31.74.254:36051)
0806 02:59:33.906631 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:33.906640 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:33.908416 (+ 1776us) server_negotiation.cc:407] Connection header received
0806 02:59:33.908785 (+ 369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:33.908794 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:33.909057 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:33.909761 (+ 704us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:33.913145 (+ 3384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:33.914541 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:33.918434 (+ 3893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:33.919870 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:33.920220 (+ 350us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:33.923712 (+ 3492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:33.923722 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:33.923729 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:33.923782 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:33.928689 (+ 4907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:33.929753 (+ 1064us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:33.929760 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:33.929763 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:33.929951 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:33.930511 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:33.930517 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:33.930521 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:33.931005 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:33.931169 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:33.931961 (+ 792us) server_negotiation.cc:299] Negotiation successful
0806 02:59:33.932237 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260806 02:59:33.933916 6662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:33.906382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:36051 (local address 127.0.0.1:46310)
0806 02:59:33.908253 (+ 1871us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:33.908290 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:33.908576 (+ 286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:33.909909 (+ 1333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:33.909915 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:33.912326 (+ 2411us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:59:33.912901 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:33.912916 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:33.914795 (+ 1879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:33.914800 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:33.918030 (+ 3230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:33.918076 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:33.920625 (+ 2549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:33.920632 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:33.921005 (+ 373us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:33.921857 (+ 852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:59:33.921886 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:59:33.923333 (+ 1447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:59:33.928879 (+ 5546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:33.928886 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:33.928893 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:33.929538 (+ 645us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:33.930137 (+ 599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:33.930142 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:33.930146 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:33.930345 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:33.931278 (+ 933us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:59:33.931284 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:59:33.931548 (+ 264us) client_negotiation.cc:769] Sending connection context
0806 02:59:33.932928 (+ 1380us) client_negotiation.cc:240] Negotiation successful
0806 02:59:33.933319 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1671,"mutex_wait_us":79,"spinlock_wait_cycles":26880,"thread_start_us":1288,"threads_started":1}
I20260806 02:59:33.936316 6568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:33.906961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48528 (local address 127.31.74.253:40869)
0806 02:59:33.907376 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:33.907386 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:33.909014 (+ 1628us) server_negotiation.cc:407] Connection header received
0806 02:59:33.910188 (+ 1174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:33.910414 (+ 226us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:33.910530 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:33.910650 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:33.912954 (+ 2304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:33.914439 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:33.918037 (+ 3598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:33.918916 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:33.919051 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:33.922750 (+ 3699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:33.922758 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:33.922762 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:33.922814 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:33.930904 (+ 8090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:33.932105 (+ 1201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:33.932111 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:33.932114 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:33.932288 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:33.933755 (+ 1467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:33.933763 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:33.933768 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:33.934838 (+ 1070us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:33.934981 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:33.935670 (+ 689us) server_negotiation.cc:299] Negotiation successful
0806 02:59:33.935950 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208}
I20260806 02:59:33.937669 6640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:33.909111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39616 (local address 127.31.74.252:35959)
0806 02:59:33.909324 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:33.909330 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:33.912182 (+ 2852us) server_negotiation.cc:407] Connection header received
0806 02:59:33.912429 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:33.912434 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:33.912523 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:33.912622 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:33.914937 (+ 2315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:33.917072 (+ 2135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:33.919754 (+ 2682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:33.920955 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:33.921079 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:33.924104 (+ 3025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:33.924111 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:33.924116 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:33.924155 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:33.928805 (+ 4650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:33.934497 (+ 5692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:33.934503 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:33.934507 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:33.934735 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:33.935194 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:33.935199 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:33.935204 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:33.935666 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:33.935839 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:33.936738 (+ 899us) server_negotiation.cc:299] Negotiation successful
0806 02:59:33.937103 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":120,"server-negotiator.queue_time_us":79}
I20260806 02:59:33.938946 6664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:33.909465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.252:35959 (local address 127.0.0.1:39616)
0806 02:59:33.912069 (+ 2604us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:33.912096 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:33.912271 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:33.913401 (+ 1130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:33.913410 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:33.914132 (+ 722us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:59:33.914754 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:33.914769 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:33.917468 (+ 2699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:33.917475 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:33.919583 (+ 2108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:33.919596 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:33.921417 (+ 1821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:33.921423 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:33.921610 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:33.922343 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:59:33.922367 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:59:33.923873 (+ 1506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:59:33.928994 (+ 5121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:33.928999 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:33.929003 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:33.934353 (+ 5350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:33.934894 (+ 541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:33.934899 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:33.934902 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:33.935055 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:33.935917 (+ 862us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:59:33.935922 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:59:33.936138 (+ 216us) client_negotiation.cc:769] Sending connection context
0806 02:59:33.938004 (+ 1866us) client_negotiation.cc:240] Negotiation successful
0806 02:59:33.938328 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2443,"thread_start_us":2011,"threads_started":1}
I20260806 02:59:33.940879 6663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:33.906685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.253:40869 (local address 127.0.0.1:48528)
0806 02:59:33.908771 (+ 2086us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:33.908790 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:33.909820 (+ 1030us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:33.910805 (+ 985us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:33.910812 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:33.912232 (+ 1420us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:59:33.912738 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:33.912757 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:33.914734 (+ 1977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:33.914739 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:33.917768 (+ 3029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:33.917784 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:33.919429 (+ 1645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:33.919437 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:33.919664 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:33.920754 (+ 1090us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:59:33.920782 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:59:33.922437 (+ 1655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:59:33.931342 (+ 8905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:33.931348 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:33.931353 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:33.931951 (+ 598us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:33.932460 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:33.932465 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:33.932468 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:33.932686 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:33.935081 (+ 2395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:59:33.935086 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:59:33.935355 (+ 269us) client_negotiation.cc:769] Sending connection context
0806 02:59:33.938491 (+ 3136us) client_negotiation.cc:240] Negotiation successful
0806 02:59:33.938942 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1934,"mutex_wait_us":39,"spinlock_wait_cycles":2304,"thread_start_us":1696,"threads_started":1}
Aug 06 02:59:33 dist-test-slave-dhph krb5kdc[5855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985170, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.74.254@KRBTEST.COM for oryx/127.31.74.252@KRBTEST.COM
Aug 06 02:59:33 dist-test-slave-dhph krb5kdc[5855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985169, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.74.193@KRBTEST.COM
I20260806 02:59:33.957646 6640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:33.941047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39630 (local address 127.31.74.252:35959)
0806 02:59:33.941263 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:33.941270 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:33.941287 (+ 17us) server_negotiation.cc:407] Connection header received
0806 02:59:33.941380 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:33.941383 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:33.941481 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:33.941607 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:33.942744 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:33.943911 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:33.945734 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:33.946259 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:33.946440 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:33.950394 (+ 3954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:33.950400 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:33.950405 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:33.950445 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:33.954398 (+ 3953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:33.955398 (+ 1000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:33.955404 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:33.955409 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:33.955538 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:33.955950 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:33.955955 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:33.955958 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:33.956451 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:33.956616 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:33.957025 (+ 409us) server_negotiation.cc:299] Negotiation successful
0806 02:59:33.957263 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260806 02:59:33.957647 6473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:33.940781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.252:35959 (local address 127.0.0.1:39630)
0806 02:59:33.941003 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:33.941027 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:33.941152 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:33.941670 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:33.941675 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:33.942201 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:59:33.942442 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:33.942456 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:33.944072 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:33.944078 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:33.945442 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:33.945460 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:33.946447 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:33.946453 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:33.946671 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:33.947334 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:59:33.947352 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:59:33.950165 (+ 2813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:59:33.954704 (+ 4539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:33.954710 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:33.954714 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:33.955275 (+ 561us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:33.955662 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:33.955667 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:33.955671 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:33.955827 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:33.956577 (+ 750us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:59:33.956583 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:59:33.956782 (+ 199us) client_negotiation.cc:769] Sending connection context
0806 02:59:33.956932 (+ 150us) client_negotiation.cc:240] Negotiation successful
0806 02:59:33.957257 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260806 02:59:33.959975 5904 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.31.74.252:35959: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "2698ead82bb0481bbc3bfee291d34fdc"
member_type: NON_VOTER
last_known_addr {
host: "127.31.74.252"
port: 35959
}
attrs {
promote: true
}
}
dest_uuid: "ec7ce3806c9d48a2bbbfacaed8423300"
cas_config_opid_index: 5
I20260806 02:59:33.960923 5904 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec7ce3806c9d48a2bbbfacaed8423300 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ec7ce3806c9d48a2bbbfacaed8423300" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36051 } } peers { permanent_uuid: "001d2d57ebfd4d6aa7d81fde49a50231" member_type: NON_VOTER last_known_addr { host: "127.31.74.253" port: 40869 } attrs { promote: true } } peers { permanent_uuid: "2698ead82bb0481bbc3bfee291d34fdc" member_type: NON_VOTER last_known_addr { host: "127.31.74.252" port: 35959 } attrs { promote: true } }
I20260806 02:59:33.962929 6476 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ec7ce3806c9d48a2bbbfacaed8423300 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 2698ead82bb0481bbc3bfee291d34fdc (127.31.74.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ec7ce3806c9d48a2bbbfacaed8423300" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36051 } } peers { permanent_uuid: "001d2d57ebfd4d6aa7d81fde49a50231" member_type: NON_VOTER last_known_addr { host: "127.31.74.253" port: 40869 } attrs { promote: true } } peers { permanent_uuid: "2698ead82bb0481bbc3bfee291d34fdc" member_type: NON_VOTER last_known_addr { host: "127.31.74.252" port: 35959 } attrs { promote: true } } }
I20260806 02:59:33.963735 6527 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 001d2d57ebfd4d6aa7d81fde49a50231 [term 1 LEARNER]: Refusing update from remote peer ec7ce3806c9d48a2bbbfacaed8423300: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260806 02:59:33.965390 6610 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2698ead82bb0481bbc3bfee291d34fdc [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:59:33.966522 6665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:33.928619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47386 (local address 127.31.74.193:46705)
0806 02:59:33.929858 (+ 1239us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:33.929868 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:33.929903 (+ 35us) server_negotiation.cc:407] Connection header received
0806 02:59:33.929979 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:33.929989 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:33.930411 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:33.932498 (+ 2087us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:33.933005 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:33.941908 (+ 8903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:33.943691 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:33.944520 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:33.944657 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:33.956336 (+ 11679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:33.956361 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:33.956385 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:33.956435 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:33.961339 (+ 4904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:33.962483 (+ 1144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:33.962493 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:33.962497 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:33.962745 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:33.963334 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:33.963339 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:33.963342 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:33.963980 (+ 638us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:33.965575 (+ 1595us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:33.965769 (+ 194us) server_negotiation.cc:299] Negotiation successful
0806 02:59:33.966061 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1037,"thread_start_us":2412,"threads_started":1}
I20260806 02:59:33.966830 6476 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.31.74.252:35959
I20260806 02:59:33.970901 6091 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:59:33.976751 6476 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ec7ce3806c9d48a2bbbfacaed8423300 [LEADER]: Connected to new peer: Peer: permanent_uuid: "001d2d57ebfd4d6aa7d81fde49a50231" member_type: NON_VOTER last_known_addr { host: "127.31.74.253" port: 40869 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 02:59:33.976802 6666 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec7ce3806c9d48a2bbbfacaed8423300 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ec7ce3806c9d48a2bbbfacaed8423300" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ec7ce3806c9d48a2bbbfacaed8423300" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36051 } } peers { permanent_uuid: "001d2d57ebfd4d6aa7d81fde49a50231" member_type: NON_VOTER last_known_addr { host: "127.31.74.253" port: 40869 } attrs { promote: true } } peers { permanent_uuid: "2698ead82bb0481bbc3bfee291d34fdc" member_type: NON_VOTER last_known_addr { host: "127.31.74.252" port: 35959 } attrs { promote: true } } }
I20260806 02:59:33.977730 6666 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec7ce3806c9d48a2bbbfacaed8423300 [sys.catalog]: This master's current role is: LEADER
I20260806 02:59:33.991215 6483 master.cc:537] Master@127.31.74.252:35959 shutting down...
I20260806 02:59:33.993527 6610 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2698ead82bb0481bbc3bfee291d34fdc [term 1 FOLLOWER]: Refusing update from remote peer ec7ce3806c9d48a2bbbfacaed8423300: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260806 02:59:33.995573 6476 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ec7ce3806c9d48a2bbbfacaed8423300 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2698ead82bb0481bbc3bfee291d34fdc" member_type: NON_VOTER last_known_addr { host: "127.31.74.252" port: 35959 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 02:59:34.010905 6589 tablet.cc:2369] T 00000000000000000000000000000000 P 2698ead82bb0481bbc3bfee291d34fdc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:59:34.018512 6483 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 2698ead82bb0481bbc3bfee291d34fdc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:59:34.019311 6483 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 2698ead82bb0481bbc3bfee291d34fdc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:59:34.019713 6483 tablet_replica.cc:330] T 00000000000000000000000000000000 P 2698ead82bb0481bbc3bfee291d34fdc: stopping tablet replica
I20260806 02:59:34.019073 6527 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 001d2d57ebfd4d6aa7d81fde49a50231 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 2698ead82bb0481bbc3bfee291d34fdc (127.31.74.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ec7ce3806c9d48a2bbbfacaed8423300" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36051 } } peers { permanent_uuid: "001d2d57ebfd4d6aa7d81fde49a50231" member_type: NON_VOTER last_known_addr { host: "127.31.74.253" port: 40869 } attrs { promote: true } } peers { permanent_uuid: "2698ead82bb0481bbc3bfee291d34fdc" member_type: NON_VOTER last_known_addr { host: "127.31.74.252" port: 35959 } attrs { promote: true } } }
I20260806 02:59:34.025303 6556 sys_catalog.cc:455] T 00000000000000000000000000000000 P 001d2d57ebfd4d6aa7d81fde49a50231 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ec7ce3806c9d48a2bbbfacaed8423300" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ec7ce3806c9d48a2bbbfacaed8423300" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36051 } } peers { permanent_uuid: "001d2d57ebfd4d6aa7d81fde49a50231" member_type: NON_VOTER last_known_addr { host: "127.31.74.253" port: 40869 } attrs { promote: true } } peers { permanent_uuid: "2698ead82bb0481bbc3bfee291d34fdc" member_type: NON_VOTER last_known_addr { host: "127.31.74.252" port: 35959 } attrs { promote: true } } }
I20260806 02:59:34.025768 6556 sys_catalog.cc:458] T 00000000000000000000000000000000 P 001d2d57ebfd4d6aa7d81fde49a50231 [sys.catalog]: This master's current role is: LEARNER
W20260806 02:59:34.027511 5888 consensus_peers.cc:487] T 00000000000000000000000000000000 P ec7ce3806c9d48a2bbbfacaed8423300 -> Peer 2698ead82bb0481bbc3bfee291d34fdc (127.31.74.252:35959): Couldn't send request to peer 2698ead82bb0481bbc3bfee291d34fdc. Status: Network error: recv got EOF from unknown peer (error 108). This is attempt 1: this message will repeat every 5th retry.
I20260806 02:59:34.027681 6667 raft_consensus.cc:1050] T 00000000000000000000000000000000 P ec7ce3806c9d48a2bbbfacaed8423300: attempting to promote NON_VOTER 001d2d57ebfd4d6aa7d81fde49a50231 to VOTER
I20260806 02:59:34.028949 6667 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec7ce3806c9d48a2bbbfacaed8423300 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ec7ce3806c9d48a2bbbfacaed8423300" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36051 } } peers { permanent_uuid: "001d2d57ebfd4d6aa7d81fde49a50231" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 40869 } attrs { promote: false } } peers { permanent_uuid: "2698ead82bb0481bbc3bfee291d34fdc" member_type: NON_VOTER last_known_addr { host: "127.31.74.252" port: 35959 } attrs { promote: true } }
I20260806 02:59:34.031332 6483 master.cc:552] Master@127.31.74.252:35959 shutdown complete.
I20260806 02:59:34.031490 6483 master_runner.cc:305] Clearing existing system tablet
I20260806 02:59:34.031953 6483 fs_manager.cc:359] Metadata directory not provided
I20260806 02:59:34.032163 6483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:59:34.032717 6473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:34.032012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.252:35959 (local address 127.0.0.1:39638)
0806 02:59:34.032404 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:34.032519 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.252:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260}
I20260806 02:59:34.033854 6527 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 001d2d57ebfd4d6aa7d81fde49a50231 [term 1 LEARNER]: Refusing update from remote peer ec7ce3806c9d48a2bbbfacaed8423300: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260806 02:59:34.034099 6473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:34.033666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.252:35959 (local address 127.0.0.1:39650)
0806 02:59:34.033842 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:34.033932 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.252:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260806 02:59:34.034382 5888 consensus_peers.cc:487] T 00000000000000000000000000000000 P ec7ce3806c9d48a2bbbfacaed8423300 -> Peer 2698ead82bb0481bbc3bfee291d34fdc (127.31.74.252:35959): Couldn't send request to peer 2698ead82bb0481bbc3bfee291d34fdc. Status: Network error: Client connection negotiation failed: client connection to 127.31.74.252:35959: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260806 02:59:34.034832 6476 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ec7ce3806c9d48a2bbbfacaed8423300 [LEADER]: Connected to new peer: Peer: permanent_uuid: "001d2d57ebfd4d6aa7d81fde49a50231" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 40869 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260806 02:59:34.036854 6483 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:59:34.038187 6667 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ec7ce3806c9d48a2bbbfacaed8423300 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 001d2d57ebfd4d6aa7d81fde49a50231 (127.31.74.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ec7ce3806c9d48a2bbbfacaed8423300" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36051 } } peers { permanent_uuid: "001d2d57ebfd4d6aa7d81fde49a50231" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 40869 } attrs { promote: false } } peers { permanent_uuid: "2698ead82bb0481bbc3bfee291d34fdc" member_type: NON_VOTER last_known_addr { host: "127.31.74.252" port: 35959 } attrs { promote: true } } }
I20260806 02:59:34.040071 6527 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 001d2d57ebfd4d6aa7d81fde49a50231 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 001d2d57ebfd4d6aa7d81fde49a50231 (127.31.74.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ec7ce3806c9d48a2bbbfacaed8423300" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36051 } } peers { permanent_uuid: "001d2d57ebfd4d6aa7d81fde49a50231" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 40869 } attrs { promote: false } } peers { permanent_uuid: "2698ead82bb0481bbc3bfee291d34fdc" member_type: NON_VOTER last_known_addr { host: "127.31.74.252" port: 35959 } attrs { promote: true } } }
I20260806 02:59:34.043071 6666 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec7ce3806c9d48a2bbbfacaed8423300 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ec7ce3806c9d48a2bbbfacaed8423300" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ec7ce3806c9d48a2bbbfacaed8423300" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36051 } } peers { permanent_uuid: "001d2d57ebfd4d6aa7d81fde49a50231" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 40869 } attrs { promote: false } } peers { permanent_uuid: "2698ead82bb0481bbc3bfee291d34fdc" member_type: NON_VOTER last_known_addr { host: "127.31.74.252" port: 35959 } attrs { promote: true } } }
I20260806 02:59:34.043340 6666 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec7ce3806c9d48a2bbbfacaed8423300 [sys.catalog]: This master's current role is: LEADER
I20260806 02:59:34.043753 6667 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec7ce3806c9d48a2bbbfacaed8423300 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ec7ce3806c9d48a2bbbfacaed8423300" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ec7ce3806c9d48a2bbbfacaed8423300" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36051 } } peers { permanent_uuid: "001d2d57ebfd4d6aa7d81fde49a50231" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 40869 } attrs { promote: false } } peers { permanent_uuid: "2698ead82bb0481bbc3bfee291d34fdc" member_type: NON_VOTER last_known_addr { host: "127.31.74.252" port: 35959 } attrs { promote: true } } }
I20260806 02:59:34.044134 6667 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec7ce3806c9d48a2bbbfacaed8423300 [sys.catalog]: This master's current role is: LEADER
I20260806 02:59:34.045122 6556 sys_catalog.cc:455] T 00000000000000000000000000000000 P 001d2d57ebfd4d6aa7d81fde49a50231 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ec7ce3806c9d48a2bbbfacaed8423300" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ec7ce3806c9d48a2bbbfacaed8423300" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36051 } } peers { permanent_uuid: "001d2d57ebfd4d6aa7d81fde49a50231" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 40869 } attrs { promote: false } } peers { permanent_uuid: "2698ead82bb0481bbc3bfee291d34fdc" member_type: NON_VOTER last_known_addr { host: "127.31.74.252" port: 35959 } attrs { promote: true } } }
I20260806 02:59:34.045403 6556 sys_catalog.cc:458] T 00000000000000000000000000000000 P 001d2d57ebfd4d6aa7d81fde49a50231 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:59:34.046989 6674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:34.047523 6483 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:59:34.048198 6483 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.003s sys 0.000s
I20260806 02:59:34.048794 6483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/master-2/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/master-2/wal
uuid: "2698ead82bb0481bbc3bfee291d34fdc"
format_stamp: "Formatted at 2026-08-06 02:59:33 on dist-test-slave-dhph"
server_key: "29426f20039b38a4f09c6c85eb1b87c8"
server_key_iv: "a22eefb151896bbaa5fb98d0c0cac467"
server_key_version: "encryptionkey@0"
I20260806 02:59:34.054797 6483 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 2698ead82bb0481bbc3bfee291d34fdc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:59:34.063645 6483 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 2698ead82bb0481bbc3bfee291d34fdc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260806 02:59:34.063983 6483 log.cc:1196] T 00000000000000000000000000000000 P 2698ead82bb0481bbc3bfee291d34fdc: Deleting WAL directory at /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260806 02:59:34.064580 6483 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 2698ead82bb0481bbc3bfee291d34fdc: Deleting consensus metadata
I20260806 02:59:34.064919 6483 master_runner.cc:315] Copying system tablet from 127.31.74.254:36051
I20260806 02:59:34.067785 6483 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 2698ead82bb0481bbc3bfee291d34fdc: tablet copy: Beginning tablet copy session from remote peer at address 127.31.74.254:36051
I20260806 02:59:34.088924 6642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:34.069749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46318 (local address 127.31.74.254:36051)
0806 02:59:34.070044 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:34.070055 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:34.070805 (+ 750us) server_negotiation.cc:407] Connection header received
0806 02:59:34.071317 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:34.071326 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:34.071421 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:34.071506 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:34.072906 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:34.074447 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:34.077077 (+ 2630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:34.077777 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:34.077898 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:34.080704 (+ 2806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:34.080712 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:34.080717 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:34.080762 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:34.084702 (+ 3940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:34.085648 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:34.085654 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:34.085659 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:34.085847 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:34.086383 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:34.086388 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:34.086392 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:34.086799 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:34.086949 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:34.088042 (+ 1093us) server_negotiation.cc:299] Negotiation successful
0806 02:59:34.088420 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260806 02:59:34.091087 6680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:34.069687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:36051 (local address 127.0.0.1:46318)
0806 02:59:34.070634 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:34.070658 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:34.070999 (+ 341us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:34.071759 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:34.071765 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:34.072413 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:59:34.072696 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:34.072709 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:34.074726 (+ 2017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:34.074732 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:34.076868 (+ 2136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:34.076886 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:34.078243 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:34.078250 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:34.078441 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:34.079284 (+ 843us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:59:34.079311 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:59:34.080435 (+ 1124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:59:34.084900 (+ 4465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:34.084906 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:34.084911 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:34.085496 (+ 585us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:34.086005 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:34.086011 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:34.086015 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:34.086231 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:34.087034 (+ 803us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:59:34.087039 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:59:34.087441 (+ 402us) client_negotiation.cc:769] Sending connection context
0806 02:59:34.089692 (+ 2251us) client_negotiation.cc:240] Negotiation successful
0806 02:59:34.090236 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"thread_start_us":458,"threads_started":1}
I20260806 02:59:34.092782 5924 tablet_copy_service.cc:140] P ec7ce3806c9d48a2bbbfacaed8423300: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 2698ead82bb0481bbc3bfee291d34fdc ({username='oryx', principal='oryx/127.31.74.252@KRBTEST.COM'} at 127.0.0.1:46318)
I20260806 02:59:34.093031 5924 tablet_copy_service.cc:161] P ec7ce3806c9d48a2bbbfacaed8423300: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 2698ead82bb0481bbc3bfee291d34fdc at {username='oryx', principal='oryx/127.31.74.252@KRBTEST.COM'} at 127.0.0.1:46318: session id = 2698ead82bb0481bbc3bfee291d34fdc-00000000000000000000000000000000
I20260806 02:59:34.097448 5924 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P ec7ce3806c9d48a2bbbfacaed8423300: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 02:59:34.099836 6483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260806 02:59:34.107692 6483 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 2698ead82bb0481bbc3bfee291d34fdc: tablet copy: Starting download of 0 data blocks...
I20260806 02:59:34.108569 6483 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 2698ead82bb0481bbc3bfee291d34fdc: tablet copy: Starting download of 1 WAL segments...
I20260806 02:59:34.112268 6483 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 2698ead82bb0481bbc3bfee291d34fdc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 02:59:34.115770 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 5965
I20260806 02:59:34.118834 5924 tablet_copy_service.cc:342] P ec7ce3806c9d48a2bbbfacaed8423300: Request end of tablet copy session 2698ead82bb0481bbc3bfee291d34fdc-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.31.74.252@KRBTEST.COM'} at 127.0.0.1:46318
I20260806 02:59:34.119027 5924 tablet_copy_service.cc:433] P ec7ce3806c9d48a2bbbfacaed8423300: ending tablet copy session 2698ead82bb0481bbc3bfee291d34fdc-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 2698ead82bb0481bbc3bfee291d34fdc
I20260806 02:59:34.131929 6483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:59:34.135025 6681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:34.139207 6682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:34.145131 5887 connection.cc:535] server connection from 127.31.74.193:58093 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260806 02:59:34.147464 6483 server_base.cc:1034] running on GCE node
W20260806 02:59:34.148139 6684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:59:34.148447 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.193:46705
--local_ip_for_outbound_sockets=127.31.74.193
--tserver_master_addrs=127.31.74.254:36051
--webserver_port=41173
--webserver_interface=127.31.74.193
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/krb5kdc/oryx_127.31.74.193.keytab
--principal=oryx/127.31.74.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:40361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.31.74.254:36051,127.31.74.253:40869
--tserver_master_addrs=127.31.74.254:36051,127.31.74.253:40869,127.31.74.252:35959 with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:59:34.148746 6483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:59:34.150009 6483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:59:34.151223 6483 hybrid_clock.cc:648] HybridClock initialized: now 1785985174151157 us; error 45 us; skew 500 ppm
I20260806 02:59:34.151604 6483 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:59:34 dist-test-slave-dhph krb5kdc[5855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985174, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.74.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:59:34.154525 6483 init.cc:377] Logged in from keytab as oryx/127.31.74.252@KRBTEST.COM (short username oryx)
I20260806 02:59:34.162876 6483 webserver.cc:460] Webserver started at http://127.31.74.252:41935/ using document root <none> and password file <none>
I20260806 02:59:34.163239 6483 fs_manager.cc:359] Metadata directory not provided
I20260806 02:59:34.163367 6483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:59:34.166042 6483 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260806 02:59:34.169411 6692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:34.170069 6483 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:59:34.170240 6483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/master-2/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/master-2/wal
uuid: "2698ead82bb0481bbc3bfee291d34fdc"
format_stamp: "Formatted at 2026-08-06 02:59:33 on dist-test-slave-dhph"
server_key: "29426f20039b38a4f09c6c85eb1b87c8"
server_key_iv: "a22eefb151896bbaa5fb98d0c0cac467"
server_key_version: "encryptionkey@0"
I20260806 02:59:34.170475 6483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:59:34.209175 6483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:59:34.213707 6483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:59:34.251134 6483 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.252:35959
I20260806 02:59:34.251204 6745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.252:35959 every 8 connection(s)
I20260806 02:59:34.252280 6483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/master-2/data/info.pb
I20260806 02:59:34.263273 6746 sys_catalog.cc:263] Verifying existing consensus state
I20260806 02:59:34.266326 6746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2698ead82bb0481bbc3bfee291d34fdc: Bootstrap starting.
I20260806 02:59:34.285419 6746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2698ead82bb0481bbc3bfee291d34fdc: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 02:59:34.285981 6746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2698ead82bb0481bbc3bfee291d34fdc: Bootstrap complete.
I20260806 02:59:34.287161 6746 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2698ead82bb0481bbc3bfee291d34fdc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ec7ce3806c9d48a2bbbfacaed8423300" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36051 } } peers { permanent_uuid: "001d2d57ebfd4d6aa7d81fde49a50231" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 40869 } attrs { promote: false } } peers { permanent_uuid: "2698ead82bb0481bbc3bfee291d34fdc" member_type: NON_VOTER last_known_addr { host: "127.31.74.252" port: 35959 } attrs { promote: true } }
I20260806 02:59:34.287431 6746 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2698ead82bb0481bbc3bfee291d34fdc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2698ead82bb0481bbc3bfee291d34fdc, State: Initialized, Role: LEARNER
I20260806 02:59:34.287703 6746 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2698ead82bb0481bbc3bfee291d34fdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ec7ce3806c9d48a2bbbfacaed8423300" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36051 } } peers { permanent_uuid: "001d2d57ebfd4d6aa7d81fde49a50231" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 40869 } attrs { promote: false } } peers { permanent_uuid: "2698ead82bb0481bbc3bfee291d34fdc" member_type: NON_VOTER last_known_addr { host: "127.31.74.252" port: 35959 } attrs { promote: true } }
I20260806 02:59:34.288578 6749 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2698ead82bb0481bbc3bfee291d34fdc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ec7ce3806c9d48a2bbbfacaed8423300" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36051 } } peers { permanent_uuid: "001d2d57ebfd4d6aa7d81fde49a50231" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 40869 } attrs { promote: false } } peers { permanent_uuid: "2698ead82bb0481bbc3bfee291d34fdc" member_type: NON_VOTER last_known_addr { host: "127.31.74.252" port: 35959 } attrs { promote: true } } }
I20260806 02:59:34.288991 6746 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2698ead82bb0481bbc3bfee291d34fdc [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:59:34.289141 6749 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2698ead82bb0481bbc3bfee291d34fdc [sys.catalog]: This master's current role is: LEARNER
I20260806 02:59:34.298274 6760 catalog_manager.cc:1259] Loaded cluster ID: 8e6a631af4aa433eb774f4779a94b441
I20260806 02:59:34.298424 6760 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2698ead82bb0481bbc3bfee291d34fdc: loading cluster ID for follower catalog manager: success
I20260806 02:59:34.302009 6760 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2698ead82bb0481bbc3bfee291d34fdc: acquiring CA information for follower catalog manager: success
I20260806 02:59:34.304495 6760 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2698ead82bb0481bbc3bfee291d34fdc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 02:59:34.440001 6473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:34.421444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.252:35959 (local address 127.0.0.1:39654)
0806 02:59:34.421937 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:34.421964 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:34.422124 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:34.431183 (+ 9059us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:34.431191 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:34.431247 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:59:34.431496 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:34.431511 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:34.434178 (+ 2667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:34.434186 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:34.436432 (+ 2246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:34.436464 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:34.438954 (+ 2490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:34.438962 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:34.439227 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:34.439275 (+ 48us) client_negotiation.cc:769] Sending connection context
0806 02:59:34.439371 (+ 96us) client_negotiation.cc:240] Negotiation successful
0806 02:59:34.439556 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331}
I20260806 02:59:34.441365 6761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:34.422682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39654 (local address 127.31.74.252:35959)
0806 02:59:34.430411 (+ 7729us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:34.430434 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:34.430484 (+ 50us) server_negotiation.cc:407] Connection header received
0806 02:59:34.430650 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:34.430658 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:34.430819 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:34.431092 (+ 273us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:59:34.431808 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:34.433833 (+ 2025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:34.436773 (+ 2940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:34.438536 (+ 1763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:34.440363 (+ 1827us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:34.440659 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:34.440743 (+ 84us) server_negotiation.cc:299] Negotiation successful
0806 02:59:34.440923 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7301,"thread_start_us":7103,"threads_started":1}
I20260806 02:59:34.442763 6720 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 2698ead82bb0481bbc3bfee291d34fdc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260806 02:59:34.443660 6749 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2698ead82bb0481bbc3bfee291d34fdc [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec7ce3806c9d48a2bbbfacaed8423300. Latest consensus state: current_term: 1 leader_uuid: "ec7ce3806c9d48a2bbbfacaed8423300" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ec7ce3806c9d48a2bbbfacaed8423300" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36051 } } peers { permanent_uuid: "001d2d57ebfd4d6aa7d81fde49a50231" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 40869 } attrs { promote: false } } peers { permanent_uuid: "2698ead82bb0481bbc3bfee291d34fdc" member_type: NON_VOTER last_known_addr { host: "127.31.74.252" port: 35959 } attrs { promote: true } } }
I20260806 02:59:34.444020 6749 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2698ead82bb0481bbc3bfee291d34fdc [sys.catalog]: This master's current role is: LEARNER
W20260806 02:59:34.638790 6689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:59:34.639155 6689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:59:34.639271 6689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:59:34.649441 6689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:59:34.649590 6689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:59:34.649732 6689 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.193
I20260806 02:59:34.662943 6689 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:40361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/krb5kdc/oryx_127.31.74.193.keytab
--principal=oryx/127.31.74.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.193:46705
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.74.193
--webserver_port=41173
--tserver_master_addrs=127.31.74.254:36051,127.31.74.253:40869,127.31.74.252:35959
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.193
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:59:34.664568 6689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:59:34.666356 6689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:59:34.684252 6766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:34.684170 6767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:34.684160 6769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:59:34.685483 6689 server_base.cc:1034] running on GCE node
I20260806 02:59:34.686309 6689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:59:34.687564 6689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:59:34.688844 6689 hybrid_clock.cc:648] HybridClock initialized: now 1785985174688768 us; error 53 us; skew 500 ppm
I20260806 02:59:34.689224 6689 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:59:34 dist-test-slave-dhph krb5kdc[5855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985174, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.74.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:59:34.693746 6689 init.cc:377] Logged in from keytab as oryx/127.31.74.193@KRBTEST.COM (short username oryx)
I20260806 02:59:34.695575 6689 webserver.cc:460] Webserver started at http://127.31.74.193:41173/ using document root <none> and password file <none>
I20260806 02:59:34.696408 6689 fs_manager.cc:359] Metadata directory not provided
I20260806 02:59:34.696601 6689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:59:34.703171 6689 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260806 02:59:34.706772 6776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:34.708392 6689 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 02:59:34.708622 6689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-0/wal
uuid: "140e7485da14480cb28af227351c865f"
format_stamp: "Formatted at 2026-08-06 02:59:30 on dist-test-slave-dhph"
server_key: "85871f41c779d879a68629bf52bc0247"
server_key_iv: "63c4b46959f031926a7f66f90a3816d1"
server_key_version: "encryptionkey@0"
I20260806 02:59:34.709925 6689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:59:34.759658 6689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:59:34.765149 6689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:59:34.765810 6689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:59:34.767474 6689 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:59:34.769639 6689 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:59:34.769874 6689 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:34.770045 6689 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:59:34.770161 6689 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:34.822254 6689 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.193:46705
I20260806 02:59:34.822455 6890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.193:46705 every 8 connection(s)
I20260806 02:59:34.824028 6689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
I20260806 02:59:34.833675 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 6689
I20260806 02:59:34.834183 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 6102
Aug 06 02:59:34 dist-test-slave-dhph krb5kdc[5855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985174, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.74.193@KRBTEST.COM for oryx/127.31.74.252@KRBTEST.COM
Aug 06 02:59:34 dist-test-slave-dhph krb5kdc[5855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985169, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.74.194@KRBTEST.COM
Aug 06 02:59:34 dist-test-slave-dhph krb5kdc[5855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985174, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.74.193@KRBTEST.COM for oryx/127.31.74.253@KRBTEST.COM
Aug 06 02:59:34 dist-test-slave-dhph krb5kdc[5855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985174, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.74.193@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM
I20260806 02:59:34.884922 6761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:34.827976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.193:46395 (local address 127.31.74.252:35959)
0806 02:59:34.828234 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:34.828240 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:34.837175 (+ 8935us) server_negotiation.cc:407] Connection header received
0806 02:59:34.849288 (+ 12113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:34.849300 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:34.849491 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:34.849600 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:34.857734 (+ 8134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:34.859510 (+ 1776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:34.864138 (+ 4628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:34.864980 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:34.865117 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:34.873516 (+ 8399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:34.873525 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:34.873530 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:34.873598 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:34.879422 (+ 5824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:34.880726 (+ 1304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:34.880735 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:34.880739 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:34.880953 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:34.881738 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:34.881746 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:34.881750 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:34.882264 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:34.882449 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:34.884243 (+ 1794us) server_negotiation.cc:299] Negotiation successful
0806 02:59:34.884476 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260806 02:59:34.888067 6894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:34.829078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.252:35959 (local address 127.31.74.193:46395)
0806 02:59:34.836902 (+ 7824us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:34.836970 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:34.848110 (+ 11140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:34.850561 (+ 2451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:34.850608 (+ 47us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:34.854459 (+ 3851us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:59:34.857356 (+ 2897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:34.857398 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:34.859834 (+ 2436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:34.859846 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:34.863772 (+ 3926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:34.863806 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:34.865509 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:34.865521 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:34.865979 (+ 458us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:34.867276 (+ 1297us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:59:34.867333 (+ 57us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:59:34.873089 (+ 5756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:59:34.879659 (+ 6570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:34.879671 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:34.879696 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:34.880539 (+ 843us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:34.881160 (+ 621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:34.881166 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:34.881171 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:34.881546 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:34.882835 (+ 1289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:59:34.882849 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:59:34.883602 (+ 753us) client_negotiation.cc:769] Sending connection context
0806 02:59:34.884794 (+ 1192us) client_negotiation.cc:240] Negotiation successful
0806 02:59:34.886185 (+ 1391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7370,"thread_start_us":6592,"threads_started":1}
I20260806 02:59:34.891237 6891 heartbeater.cc:347] Connected to a master server at 127.31.74.252:35959
I20260806 02:59:34.891645 6891 heartbeater.cc:464] Registering TS with master...
I20260806 02:59:34.892920 6891 heartbeater.cc:511] Master 127.31.74.252:35959 requested a full tablet report, sending...
I20260806 02:59:34.894344 6900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:34.837939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.193:44403 (local address 127.31.74.253:40869)
0806 02:59:34.856821 (+ 18882us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:34.856827 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:34.865240 (+ 8413us) server_negotiation.cc:407] Connection header received
0806 02:59:34.865681 (+ 441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:34.865692 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:34.865851 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:34.865980 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:34.868532 (+ 2552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:34.870289 (+ 1757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:34.875019 (+ 4730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:34.876286 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:34.876450 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:34.883717 (+ 7267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:34.883726 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:34.883732 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:34.883800 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:34.888772 (+ 4972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:34.890299 (+ 1527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:34.890307 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:34.890311 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:34.890719 (+ 408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:34.891617 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:34.891623 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:34.891627 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:34.892241 (+ 614us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:34.892557 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:34.893591 (+ 1034us) server_negotiation.cc:299] Negotiation successful
0806 02:59:34.893865 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18748,"thread_start_us":18885,"threads_started":1}
I20260806 02:59:34.895622 6899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:34.836868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36418 (local address 127.31.74.194:41805)
0806 02:59:34.859665 (+ 22797us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:34.859682 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:34.859733 (+ 51us) server_negotiation.cc:407] Connection header received
0806 02:59:34.859846 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:34.859862 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:34.860244 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:34.865030 (+ 4786us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:34.865407 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:34.867452 (+ 2045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:34.869992 (+ 2540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:34.870837 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:34.871009 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:34.882633 (+ 11624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:34.882659 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:34.882683 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:34.882734 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:34.888153 (+ 5419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:34.890781 (+ 2628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:34.890795 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:34.890801 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:34.891202 (+ 401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:34.891912 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:34.891918 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:34.891921 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:34.892862 (+ 941us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:34.894645 (+ 1783us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:34.894851 (+ 206us) server_negotiation.cc:299] Negotiation successful
0806 02:59:34.895163 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21922,"thread_start_us":21702,"threads_started":1}
I20260806 02:59:34.896047 6901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:34.844042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.253:40869 (local address 127.31.74.193:44403)
0806 02:59:34.865075 (+ 21033us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:34.865106 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:34.865433 (+ 327us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:34.866536 (+ 1103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:34.866546 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:34.867726 (+ 1180us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:59:34.868227 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:34.868247 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:34.871666 (+ 3419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:34.871757 (+ 91us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:34.874728 (+ 2971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:34.874750 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:34.876860 (+ 2110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:34.876870 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:34.877100 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:34.878347 (+ 1247us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:59:34.878561 (+ 214us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:59:34.883426 (+ 4865us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:59:34.889020 (+ 5594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:34.889025 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:34.889030 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:34.889959 (+ 929us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:34.890924 (+ 965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:34.890930 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:34.890934 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:34.891320 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:34.892741 (+ 1421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:59:34.892865 (+ 124us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:59:34.893242 (+ 377us) client_negotiation.cc:769] Sending connection context
0806 02:59:34.894962 (+ 1720us) client_negotiation.cc:240] Negotiation successful
0806 02:59:34.895298 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20794,"thread_start_us":20460,"threads_started":1}
I20260806 02:59:34.896100 6709 ts_manager.cc:194] Registered new tserver with Master: 140e7485da14480cb28af227351c865f (127.31.74.193:46705)
I20260806 02:59:34.896628 6896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:34.830504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:36051 (local address 127.31.74.193:44021)
0806 02:59:34.852794 (+ 22290us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:34.852938 (+ 144us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:34.853174 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:34.872253 (+ 19079us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:34.872258 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:34.873177 (+ 919us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:59:34.873633 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:34.873649 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:34.876724 (+ 3075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:34.876729 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:34.878870 (+ 2141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:34.878886 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:34.880497 (+ 1611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:34.880509 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:34.880781 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:34.881546 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:59:34.881578 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:59:34.886488 (+ 4910us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:59:34.892397 (+ 5909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:34.892402 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:34.892405 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:34.893183 (+ 778us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:34.893772 (+ 589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:34.893777 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:34.893781 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:34.893978 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:34.895023 (+ 1045us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:59:34.895028 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:59:34.895227 (+ 199us) client_negotiation.cc:769] Sending connection context
0806 02:59:34.895371 (+ 144us) client_negotiation.cc:240] Negotiation successful
0806 02:59:34.895732 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21557,"thread_start_us":21649,"threads_started":1}
I20260806 02:59:34.897305 6228 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:59:34.897732 6895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:34.830507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.193:44021 (local address 127.31.74.254:36051)
0806 02:59:34.871834 (+ 41327us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:34.871844 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:34.871870 (+ 26us) server_negotiation.cc:407] Connection header received
0806 02:59:34.871951 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:34.871958 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:34.872081 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:34.872316 (+ 235us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:34.874892 (+ 2576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:34.876529 (+ 1637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:34.879103 (+ 2574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:34.880160 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:34.881166 (+ 1006us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:34.886890 (+ 5724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:34.886899 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:34.886906 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:34.886960 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:34.892150 (+ 5190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:34.893387 (+ 1237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:34.893393 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:34.893396 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:34.893597 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:34.894130 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:34.894136 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:34.894142 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:34.894839 (+ 697us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:34.896921 (+ 2082us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:34.897079 (+ 158us) server_negotiation.cc:299] Negotiation successful
0806 02:59:34.897368 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41111,"thread_start_us":40863,"threads_started":1}
I20260806 02:59:34.898332 6893 heartbeater.cc:347] Connected to a master server at 127.31.74.253:40869
I20260806 02:59:34.898547 6893 heartbeater.cc:464] Registering TS with master...
I20260806 02:59:34.899325 6893 heartbeater.cc:511] Master 127.31.74.253:40869 requested a full tablet report, sending...
I20260806 02:59:34.900132 6892 heartbeater.cc:347] Connected to a master server at 127.31.74.254:36051
I20260806 02:59:34.900326 6892 heartbeater.cc:464] Registering TS with master...
I20260806 02:59:34.901039 6892 heartbeater.cc:511] Master 127.31.74.254:36051 requested a full tablet report, sending...
I20260806 02:59:34.901856 6517 ts_manager.cc:194] Registered new tserver with Master: 140e7485da14480cb28af227351c865f (127.31.74.193:46705)
I20260806 02:59:34.902889 5904 ts_manager.cc:194] Re-registered known tserver with Master: 140e7485da14480cb28af227351c865f (127.31.74.193:46705)
I20260806 02:59:34.904734 5904 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.74.193@KRBTEST.COM'} at 127.31.74.193:44021
I20260806 02:59:35.000560 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 6102
I20260806 02:59:35.024119 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.194:41805
--local_ip_for_outbound_sockets=127.31.74.194
--tserver_master_addrs=127.31.74.254:36051
--webserver_port=46785
--webserver_interface=127.31.74.194
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/krb5kdc/oryx_127.31.74.194.keytab
--principal=oryx/127.31.74.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:40361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.31.74.254:36051,127.31.74.253:40869
--tserver_master_addrs=127.31.74.254:36051,127.31.74.253:40869,127.31.74.252:35959 with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:59:35.040789 6904 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec7ce3806c9d48a2bbbfacaed8423300 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ec7ce3806c9d48a2bbbfacaed8423300" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ec7ce3806c9d48a2bbbfacaed8423300" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36051 } } peers { permanent_uuid: "001d2d57ebfd4d6aa7d81fde49a50231" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 40869 } attrs { promote: false } } peers { permanent_uuid: "2698ead82bb0481bbc3bfee291d34fdc" member_type: NON_VOTER last_known_addr { host: "127.31.74.252" port: 35959 } attrs { promote: true } } }
I20260806 02:59:35.041280 6904 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec7ce3806c9d48a2bbbfacaed8423300 [sys.catalog]: This master's current role is: LEADER
I20260806 02:59:35.042671 6906 raft_consensus.cc:1050] T 00000000000000000000000000000000 P ec7ce3806c9d48a2bbbfacaed8423300: attempting to promote NON_VOTER 2698ead82bb0481bbc3bfee291d34fdc to VOTER
I20260806 02:59:35.043287 6906 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec7ce3806c9d48a2bbbfacaed8423300 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ec7ce3806c9d48a2bbbfacaed8423300" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36051 } } peers { permanent_uuid: "001d2d57ebfd4d6aa7d81fde49a50231" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 40869 } attrs { promote: false } } peers { permanent_uuid: "2698ead82bb0481bbc3bfee291d34fdc" member_type: VOTER last_known_addr { host: "127.31.74.252" port: 35959 } attrs { promote: false } }
I20260806 02:59:35.045635 6720 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2698ead82bb0481bbc3bfee291d34fdc [term 1 LEARNER]: Refusing update from remote peer ec7ce3806c9d48a2bbbfacaed8423300: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260806 02:59:35.045993 6527 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 001d2d57ebfd4d6aa7d81fde49a50231 [term 1 FOLLOWER]: Refusing update from remote peer ec7ce3806c9d48a2bbbfacaed8423300: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260806 02:59:35.046334 6906 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ec7ce3806c9d48a2bbbfacaed8423300 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2698ead82bb0481bbc3bfee291d34fdc" member_type: VOTER last_known_addr { host: "127.31.74.252" port: 35959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260806 02:59:35.046806 6904 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ec7ce3806c9d48a2bbbfacaed8423300 [LEADER]: Connected to new peer: Peer: permanent_uuid: "001d2d57ebfd4d6aa7d81fde49a50231" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 40869 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260806 02:59:35.050209 6904 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ec7ce3806c9d48a2bbbfacaed8423300 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2698ead82bb0481bbc3bfee291d34fdc (127.31.74.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ec7ce3806c9d48a2bbbfacaed8423300" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36051 } } peers { permanent_uuid: "001d2d57ebfd4d6aa7d81fde49a50231" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 40869 } attrs { promote: false } } peers { permanent_uuid: "2698ead82bb0481bbc3bfee291d34fdc" member_type: VOTER last_known_addr { host: "127.31.74.252" port: 35959 } attrs { promote: false } } }
I20260806 02:59:35.053380 6720 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2698ead82bb0481bbc3bfee291d34fdc [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2698ead82bb0481bbc3bfee291d34fdc (127.31.74.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ec7ce3806c9d48a2bbbfacaed8423300" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36051 } } peers { permanent_uuid: "001d2d57ebfd4d6aa7d81fde49a50231" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 40869 } attrs { promote: false } } peers { permanent_uuid: "2698ead82bb0481bbc3bfee291d34fdc" member_type: VOTER last_known_addr { host: "127.31.74.252" port: 35959 } attrs { promote: false } } }
I20260806 02:59:35.054126 6904 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec7ce3806c9d48a2bbbfacaed8423300 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ec7ce3806c9d48a2bbbfacaed8423300" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ec7ce3806c9d48a2bbbfacaed8423300" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36051 } } peers { permanent_uuid: "001d2d57ebfd4d6aa7d81fde49a50231" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 40869 } attrs { promote: false } } peers { permanent_uuid: "2698ead82bb0481bbc3bfee291d34fdc" member_type: VOTER last_known_addr { host: "127.31.74.252" port: 35959 } attrs { promote: false } } }
I20260806 02:59:35.054356 6904 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec7ce3806c9d48a2bbbfacaed8423300 [sys.catalog]: This master's current role is: LEADER
I20260806 02:59:35.054644 6906 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec7ce3806c9d48a2bbbfacaed8423300 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ec7ce3806c9d48a2bbbfacaed8423300" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ec7ce3806c9d48a2bbbfacaed8423300" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36051 } } peers { permanent_uuid: "001d2d57ebfd4d6aa7d81fde49a50231" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 40869 } attrs { promote: false } } peers { permanent_uuid: "2698ead82bb0481bbc3bfee291d34fdc" member_type: VOTER last_known_addr { host: "127.31.74.252" port: 35959 } attrs { promote: false } } }
I20260806 02:59:35.054926 6906 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec7ce3806c9d48a2bbbfacaed8423300 [sys.catalog]: This master's current role is: LEADER
I20260806 02:59:35.055158 6527 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 001d2d57ebfd4d6aa7d81fde49a50231 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2698ead82bb0481bbc3bfee291d34fdc (127.31.74.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ec7ce3806c9d48a2bbbfacaed8423300" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36051 } } peers { permanent_uuid: "001d2d57ebfd4d6aa7d81fde49a50231" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 40869 } attrs { promote: false } } peers { permanent_uuid: "2698ead82bb0481bbc3bfee291d34fdc" member_type: VOTER last_known_addr { host: "127.31.74.252" port: 35959 } attrs { promote: false } } }
I20260806 02:59:35.058363 6904 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec7ce3806c9d48a2bbbfacaed8423300 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ec7ce3806c9d48a2bbbfacaed8423300" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ec7ce3806c9d48a2bbbfacaed8423300" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36051 } } peers { permanent_uuid: "001d2d57ebfd4d6aa7d81fde49a50231" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 40869 } attrs { promote: false } } peers { permanent_uuid: "2698ead82bb0481bbc3bfee291d34fdc" member_type: VOTER last_known_addr { host: "127.31.74.252" port: 35959 } attrs { promote: false } } }
I20260806 02:59:35.058734 6904 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec7ce3806c9d48a2bbbfacaed8423300 [sys.catalog]: This master's current role is: LEADER
I20260806 02:59:35.059494 6911 sys_catalog.cc:455] T 00000000000000000000000000000000 P 001d2d57ebfd4d6aa7d81fde49a50231 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ec7ce3806c9d48a2bbbfacaed8423300" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ec7ce3806c9d48a2bbbfacaed8423300" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36051 } } peers { permanent_uuid: "001d2d57ebfd4d6aa7d81fde49a50231" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 40869 } attrs { promote: false } } peers { permanent_uuid: "2698ead82bb0481bbc3bfee291d34fdc" member_type: VOTER last_known_addr { host: "127.31.74.252" port: 35959 } attrs { promote: false } } }
I20260806 02:59:35.060295 6911 sys_catalog.cc:458] T 00000000000000000000000000000000 P 001d2d57ebfd4d6aa7d81fde49a50231 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:59:35.060885 6912 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2698ead82bb0481bbc3bfee291d34fdc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ec7ce3806c9d48a2bbbfacaed8423300" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ec7ce3806c9d48a2bbbfacaed8423300" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36051 } } peers { permanent_uuid: "001d2d57ebfd4d6aa7d81fde49a50231" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 40869 } attrs { promote: false } } peers { permanent_uuid: "2698ead82bb0481bbc3bfee291d34fdc" member_type: VOTER last_known_addr { host: "127.31.74.252" port: 35959 } attrs { promote: false } } }
I20260806 02:59:35.061249 6912 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2698ead82bb0481bbc3bfee291d34fdc [sys.catalog]: This master's current role is: FOLLOWER
W20260806 02:59:35.440004 6903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:59:35.440410 6903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:59:35.440620 6903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:59:35.450100 6903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:59:35.450295 6903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:59:35.450520 6903 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.194
I20260806 02:59:35.462138 6903 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:40361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/krb5kdc/oryx_127.31.74.194.keytab
--principal=oryx/127.31.74.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.194:41805
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.74.194
--webserver_port=46785
--tserver_master_addrs=127.31.74.254:36051,127.31.74.253:40869,127.31.74.252:35959
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.194
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:59:35.463651 6903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:59:35.464960 6903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:59:35.481160 6918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:35.481092 6920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:35.481480 6917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:59:35.484071 6903 server_base.cc:1034] running on GCE node
I20260806 02:59:35.484791 6903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:59:35.485890 6903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:59:35.487190 6903 hybrid_clock.cc:648] HybridClock initialized: now 1785985175487087 us; error 71 us; skew 500 ppm
I20260806 02:59:35.487587 6903 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:59:35 dist-test-slave-dhph krb5kdc[5855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985175, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.74.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:59:35.491328 6903 init.cc:377] Logged in from keytab as oryx/127.31.74.194@KRBTEST.COM (short username oryx)
I20260806 02:59:35.493062 6903 webserver.cc:460] Webserver started at http://127.31.74.194:46785/ using document root <none> and password file <none>
I20260806 02:59:35.493819 6903 fs_manager.cc:359] Metadata directory not provided
I20260806 02:59:35.493974 6903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:59:35.499966 6903 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 02:59:35.503455 6927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:35.505501 6903 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260806 02:59:35.505714 6903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-1/wal
uuid: "d45fe0ed8f6b4ddb9b3f31f14ba18ace"
format_stamp: "Formatted at 2026-08-06 02:59:31 on dist-test-slave-dhph"
server_key: "164e3c0eaf557d7fcd84b519269ce02b"
server_key_iv: "b8cc74321680930a85b35a3aaf6a1740"
server_key_version: "encryptionkey@0"
I20260806 02:59:35.506951 6903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:59:35.534154 6903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:59:35.539611 6903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:59:35.539969 6903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:59:35.541109 6903 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:59:35.542734 6903 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:59:35.542824 6903 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:35.542959 6903 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:59:35.543061 6903 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:35.585093 6903 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.194:41805
I20260806 02:59:35.585189 7044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.194:41805 every 8 connection(s)
I20260806 02:59:35.586655 6903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
I20260806 02:59:35.591610 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 6903
I20260806 02:59:35.592151 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 6239
Aug 06 02:59:35 dist-test-slave-dhph krb5kdc[5855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985169, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.74.195@KRBTEST.COM
Aug 06 02:59:35 dist-test-slave-dhph krb5kdc[5855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985175, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.74.194@KRBTEST.COM for oryx/127.31.74.252@KRBTEST.COM
Aug 06 02:59:35 dist-test-slave-dhph krb5kdc[5855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985175, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.74.194@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM
Aug 06 02:59:35 dist-test-slave-dhph krb5kdc[5855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985175, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.74.194@KRBTEST.COM for oryx/127.31.74.253@KRBTEST.COM
I20260806 02:59:35.641340 7054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:35.593629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43394 (local address 127.31.74.195:41485)
0806 02:59:35.607195 (+ 13566us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:35.607207 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:35.607256 (+ 49us) server_negotiation.cc:407] Connection header received
0806 02:59:35.607374 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:35.607388 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:35.607898 (+ 510us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:35.608155 (+ 257us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:35.612662 (+ 4507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:35.615063 (+ 2401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:35.617928 (+ 2865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:35.619345 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:35.619498 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:35.628216 (+ 8718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:35.628236 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:35.628256 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:35.628315 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:35.636340 (+ 8025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:35.637450 (+ 1110us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:35.637461 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:35.637466 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:35.637686 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:35.638208 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:35.638213 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:35.638217 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:35.638772 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:35.639095 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:35.640476 (+ 1381us) server_negotiation.cc:299] Negotiation successful
0806 02:59:35.640864 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13356,"thread_start_us":13178,"threads_started":1}
I20260806 02:59:35.643260 6366 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:59:35.644894 7049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:35.589952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.194:44637 (local address 127.31.74.254:36051)
0806 02:59:35.600159 (+ 10207us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:35.600171 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:35.607741 (+ 7570us) server_negotiation.cc:407] Connection header received
0806 02:59:35.619771 (+ 12030us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:35.619781 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:35.619921 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:35.620048 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:35.623147 (+ 3099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:35.624428 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:35.628096 (+ 3668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:35.628817 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:35.628968 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:35.636595 (+ 7627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:35.636604 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:35.636608 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:35.636656 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:35.640546 (+ 3890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:35.641766 (+ 1220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:35.641772 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:35.641775 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:35.641947 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:35.642421 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:35.642425 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:35.642428 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:35.642818 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:35.642933 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:35.643868 (+ 935us) server_negotiation.cc:299] Negotiation successful
0806 02:59:35.644170 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10003,"thread_start_us":9844,"threads_started":1}
I20260806 02:59:35.646787 7052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:35.590745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:36051 (local address 127.31.74.194:44637)
0806 02:59:35.607502 (+ 16757us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:35.607575 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:35.619554 (+ 11979us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:35.620704 (+ 1150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:35.620713 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:35.621517 (+ 804us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:59:35.621950 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:35.621969 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:35.624641 (+ 2672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:35.624654 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:35.627915 (+ 3261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:35.627932 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:35.629099 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:35.629116 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:35.629461 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:35.631111 (+ 1650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:59:35.631144 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:59:35.636300 (+ 5156us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:59:35.640740 (+ 4440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:35.640745 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:35.640751 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:35.641611 (+ 860us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:35.642093 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:35.642097 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:35.642100 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:35.642282 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:35.643011 (+ 729us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:59:35.643016 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:59:35.643417 (+ 401us) client_negotiation.cc:769] Sending connection context
0806 02:59:35.645267 (+ 1850us) client_negotiation.cc:240] Negotiation successful
0806 02:59:35.645492 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16367,"mutex_wait_us":60,"thread_start_us":16085,"threads_started":1}
I20260806 02:59:35.646967 7055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:35.612094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.194:40155 (local address 127.31.74.253:40869)
0806 02:59:35.623454 (+ 11360us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:35.623462 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:35.623483 (+ 21us) server_negotiation.cc:407] Connection header received
0806 02:59:35.623546 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:35.623552 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:35.623649 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:35.623798 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:35.625544 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:35.627008 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:35.629917 (+ 2909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:35.630725 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:35.630884 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:35.637022 (+ 6138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:35.637030 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:35.637036 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:35.637090 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:35.641304 (+ 4214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:35.642225 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:35.642230 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:35.642234 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:35.642432 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:35.643056 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:35.643061 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:35.643064 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:35.643643 (+ 579us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:35.645807 (+ 2164us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:35.645992 (+ 185us) server_negotiation.cc:299] Negotiation successful
0806 02:59:35.646356 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11216,"thread_start_us":11303,"threads_started":1}
I20260806 02:59:35.647496 7056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:35.618419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.253:40869 (local address 127.31.74.194:40155)
0806 02:59:35.622746 (+ 4327us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:35.622788 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:35.622951 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:35.623983 (+ 1032us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:35.623987 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:35.624500 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:59:35.625186 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:35.625209 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:35.627216 (+ 2007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:35.627222 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:35.628735 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:35.628748 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:35.630983 (+ 2235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:35.630990 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:35.631168 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:35.633574 (+ 2406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:59:35.633600 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:59:35.636726 (+ 3126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:59:35.641478 (+ 4752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:35.641484 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:35.641487 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:35.642052 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:35.642563 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:35.642568 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:35.642572 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:35.642779 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:35.643854 (+ 1075us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:59:35.643863 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:59:35.644125 (+ 262us) client_negotiation.cc:769] Sending connection context
0806 02:59:35.644371 (+ 246us) client_negotiation.cc:240] Negotiation successful
0806 02:59:35.645003 (+ 632us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4114,"mutex_wait_us":4,"spinlock_wait_cycles":34560,"thread_start_us":3926,"threads_started":1}
I20260806 02:59:35.648432 7046 heartbeater.cc:347] Connected to a master server at 127.31.74.254:36051
I20260806 02:59:35.648837 7046 heartbeater.cc:464] Registering TS with master...
I20260806 02:59:35.649681 7050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:35.590192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.194:46679 (local address 127.31.74.252:35959)
0806 02:59:35.603100 (+ 12908us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:35.603111 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:35.611871 (+ 8760us) server_negotiation.cc:407] Connection header received
0806 02:59:35.613841 (+ 1970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:35.613852 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:35.613979 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:35.614096 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:35.621631 (+ 7535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:35.622928 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:35.629096 (+ 6168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:35.629860 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:35.629983 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:35.636133 (+ 6150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:35.636141 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:35.636146 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:35.636193 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:35.640299 (+ 4106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:35.641625 (+ 1326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:35.641632 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:35.641635 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:35.641783 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:35.642375 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:35.642380 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:35.642383 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:35.644992 (+ 2609us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:35.645164 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:35.649048 (+ 3884us) server_negotiation.cc:299] Negotiation successful
0806 02:59:35.649306 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12697,"thread_start_us":12884,"threads_started":1}
I20260806 02:59:35.649889 7046 heartbeater.cc:511] Master 127.31.74.254:36051 requested a full tablet report, sending...
I20260806 02:59:35.649927 7051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:35.590794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.252:35959 (local address 127.31.74.194:46679)
0806 02:59:35.611736 (+ 20942us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:35.611761 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:35.613339 (+ 1578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:35.614955 (+ 1616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:35.614971 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:35.617194 (+ 2223us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:59:35.619256 (+ 2062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:35.619313 (+ 57us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:35.624742 (+ 5429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:35.624750 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:35.627907 (+ 3157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:35.627923 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:35.630292 (+ 2369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:35.630299 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:35.630500 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:35.631799 (+ 1299us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:59:35.631827 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:59:35.635801 (+ 3974us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:59:35.640517 (+ 4716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:35.640528 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:35.640548 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:35.641478 (+ 930us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:35.641961 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:35.641965 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:35.641968 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:35.642231 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:35.648244 (+ 6013us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:59:35.648249 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:59:35.648508 (+ 259us) client_negotiation.cc:769] Sending connection context
0806 02:59:35.648823 (+ 315us) client_negotiation.cc:240] Negotiation successful
0806 02:59:35.649160 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20763,"mutex_wait_us":202,"thread_start_us":20474,"threads_started":1}
I20260806 02:59:35.650791 7047 heartbeater.cc:347] Connected to a master server at 127.31.74.253:40869
I20260806 02:59:35.650946 7047 heartbeater.cc:464] Registering TS with master...
I20260806 02:59:35.651427 7047 heartbeater.cc:511] Master 127.31.74.253:40869 requested a full tablet report, sending...
I20260806 02:59:35.652451 7045 heartbeater.cc:347] Connected to a master server at 127.31.74.252:35959
I20260806 02:59:35.652580 7045 heartbeater.cc:464] Registering TS with master...
I20260806 02:59:35.652611 5904 ts_manager.cc:194] Re-registered known tserver with Master: d45fe0ed8f6b4ddb9b3f31f14ba18ace (127.31.74.194:41805)
I20260806 02:59:35.653041 7045 heartbeater.cc:511] Master 127.31.74.252:35959 requested a full tablet report, sending...
I20260806 02:59:35.653290 6517 ts_manager.cc:194] Registered new tserver with Master: d45fe0ed8f6b4ddb9b3f31f14ba18ace (127.31.74.194:41805)
I20260806 02:59:35.653772 5904 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.74.194@KRBTEST.COM'} at 127.31.74.194:44637
I20260806 02:59:35.654318 6709 ts_manager.cc:194] Registered new tserver with Master: d45fe0ed8f6b4ddb9b3f31f14ba18ace (127.31.74.194:41805)
I20260806 02:59:35.744846 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 6239
I20260806 02:59:35.767046 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.195:41485
--local_ip_for_outbound_sockets=127.31.74.195
--tserver_master_addrs=127.31.74.254:36051
--webserver_port=35761
--webserver_interface=127.31.74.195
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/krb5kdc/oryx_127.31.74.195.keytab
--principal=oryx/127.31.74.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:40361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.31.74.254:36051,127.31.74.253:40869
--tserver_master_addrs=127.31.74.254:36051,127.31.74.253:40869,127.31.74.252:35959 with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:59:35.909022 6892 heartbeater.cc:503] Master 127.31.74.254:36051 was elected leader, sending a full tablet report...
W20260806 02:59:36.179607 7058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:59:36.179998 7058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:59:36.180204 7058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:59:36.189384 7058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:59:36.189574 7058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:59:36.189836 7058 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.195
I20260806 02:59:36.201623 7058 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:40361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/krb5kdc/oryx_127.31.74.195.keytab
--principal=oryx/127.31.74.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.195:41485
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.74.195
--webserver_port=35761
--tserver_master_addrs=127.31.74.254:36051,127.31.74.253:40869,127.31.74.252:35959
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.195
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:59:36.203289 7058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:59:36.205082 7058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:59:36.221593 7063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:36.221657 7064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:36.221628 7066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:59:36.222688 7058 server_base.cc:1034] running on GCE node
I20260806 02:59:36.223450 7058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:59:36.224525 7058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:59:36.225759 7058 hybrid_clock.cc:648] HybridClock initialized: now 1785985176225656 us; error 85 us; skew 500 ppm
I20260806 02:59:36.226285 7058 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:59:36 dist-test-slave-dhph krb5kdc[5855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985176, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.74.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:59:36.230437 7058 init.cc:377] Logged in from keytab as oryx/127.31.74.195@KRBTEST.COM (short username oryx)
I20260806 02:59:36.232404 7058 webserver.cc:460] Webserver started at http://127.31.74.195:35761/ using document root <none> and password file <none>
I20260806 02:59:36.233219 7058 fs_manager.cc:359] Metadata directory not provided
I20260806 02:59:36.233448 7058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:59:36.240640 7058 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 02:59:36.244249 7073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:36.245918 7058 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 02:59:36.246104 7058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-2/wal
uuid: "7ed1c497e3454a6eb54f4bcfda8552de"
format_stamp: "Formatted at 2026-08-06 02:59:32 on dist-test-slave-dhph"
server_key: "880492327dfab0c8727b0d7726782321"
server_key_iv: "8458138521a92c973026a2198a69bb5f"
server_key_version: "encryptionkey@0"
I20260806 02:59:36.247567 7058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:59:36.265636 7058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:59:36.271703 7058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:59:36.272140 7058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:59:36.273324 7058 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:59:36.275295 7058 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:59:36.275410 7058 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:36.275504 7058 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:59:36.275626 7058 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:36.318658 7058 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.195:41485
I20260806 02:59:36.318756 7188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.195:41485 every 8 connection(s)
I20260806 02:59:36.320221 7058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
I20260806 02:59:36.330607 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 7058
Aug 06 02:59:36 dist-test-slave-dhph krb5kdc[5855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985176, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.74.195@KRBTEST.COM for oryx/127.31.74.253@KRBTEST.COM
Aug 06 02:59:36 dist-test-slave-dhph krb5kdc[5855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985176, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.74.195@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM
Aug 06 02:59:36 dist-test-slave-dhph krb5kdc[5855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985176, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.74.195@KRBTEST.COM for oryx/127.31.74.252@KRBTEST.COM
I20260806 02:59:36.385304 7194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:36.323712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.195:36679 (local address 127.31.74.253:40869)
0806 02:59:36.336168 (+ 12456us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:36.336178 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:36.337816 (+ 1638us) server_negotiation.cc:407] Connection header received
0806 02:59:36.341669 (+ 3853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:36.341679 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:36.341907 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:36.342010 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:36.345156 (+ 3146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:36.346599 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:36.351733 (+ 5134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:36.353190 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:36.353347 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:36.368315 (+ 14968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:36.368328 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:36.368338 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:36.368402 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:36.376541 (+ 8139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:36.378843 (+ 2302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:36.378850 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:36.378855 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:36.379090 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:36.383104 (+ 4014us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:36.383110 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:36.383113 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:36.383539 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:36.383678 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:36.384580 (+ 902us) server_negotiation.cc:299] Negotiation successful
0806 02:59:36.384893 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12245,"thread_start_us":12481,"threads_started":1}
I20260806 02:59:36.385964 7196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:36.324455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:36051 (local address 127.31.74.195:60921)
0806 02:59:36.350086 (+ 25631us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:36.350117 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:36.351691 (+ 1574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:36.354155 (+ 2464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:36.354165 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:36.357046 (+ 2881us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:59:36.357721 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:36.357744 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:36.360284 (+ 2540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:36.360291 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:36.363461 (+ 3170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:36.363483 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:36.365344 (+ 1861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:36.365354 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:36.365612 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:36.366882 (+ 1270us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:59:36.366915 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:59:36.371335 (+ 4420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:59:36.378410 (+ 7075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:36.378419 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:36.378426 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:36.379664 (+ 1238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:36.380845 (+ 1181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:36.380851 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:36.380856 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:36.381194 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:36.382561 (+ 1367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:59:36.382571 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:59:36.383178 (+ 607us) client_negotiation.cc:769] Sending connection context
0806 02:59:36.383507 (+ 329us) client_negotiation.cc:240] Negotiation successful
0806 02:59:36.384420 (+ 913us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25360,"mutex_wait_us":40,"spinlock_wait_cycles":852096,"thread_start_us":24816,"threads_started":1}
I20260806 02:59:36.386950 7195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:36.324455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.253:40869 (local address 127.31.74.195:36679)
0806 02:59:36.337525 (+ 13070us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:36.337616 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:36.341176 (+ 3560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:36.342771 (+ 1595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:36.342789 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:36.343822 (+ 1033us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:59:36.344876 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:36.344906 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:36.346934 (+ 2028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:36.346942 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:36.350948 (+ 4006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:36.350978 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:36.354603 (+ 3625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:36.354612 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:36.355499 (+ 887us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:36.357411 (+ 1912us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:59:36.357628 (+ 217us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:59:36.366383 (+ 8755us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:59:36.377075 (+ 10692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:36.377098 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:36.377125 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:36.377901 (+ 776us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:36.379245 (+ 1344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:36.379249 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:36.379251 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:36.380068 (+ 817us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:36.384094 (+ 4026us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:59:36.384109 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:59:36.384285 (+ 176us) client_negotiation.cc:769] Sending connection context
0806 02:59:36.385833 (+ 1548us) client_negotiation.cc:240] Negotiation successful
0806 02:59:36.386447 (+ 614us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12638,"thread_start_us":12344,"threads_started":1}
I20260806 02:59:36.387903 7193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:36.323603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.195:60921 (local address 127.31.74.254:36051)
0806 02:59:36.336740 (+ 13137us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:36.336749 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:36.350251 (+ 13502us) server_negotiation.cc:407] Connection header received
0806 02:59:36.353743 (+ 3492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:36.353754 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:36.353921 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:36.354079 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:36.357988 (+ 3909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:36.359578 (+ 1590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:36.363708 (+ 4130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:36.364683 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:36.364840 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:36.372340 (+ 7500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:36.372351 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:36.372358 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:36.372417 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:36.378245 (+ 5828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:36.379998 (+ 1753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:36.380005 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:36.380009 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:36.380571 (+ 562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:36.381507 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:36.381514 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:36.381519 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:36.382273 (+ 754us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:36.382459 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:36.386762 (+ 4303us) server_negotiation.cc:299] Negotiation successful
0806 02:59:36.387051 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12965,"thread_start_us":12862,"threads_started":1}
I20260806 02:59:36.389496 7190 heartbeater.cc:347] Connected to a master server at 127.31.74.254:36051
I20260806 02:59:36.389900 7190 heartbeater.cc:464] Registering TS with master...
I20260806 02:59:36.391054 7190 heartbeater.cc:511] Master 127.31.74.254:36051 requested a full tablet report, sending...
I20260806 02:59:36.392138 7191 heartbeater.cc:347] Connected to a master server at 127.31.74.253:40869
I20260806 02:59:36.392298 7191 heartbeater.cc:464] Registering TS with master...
I20260806 02:59:36.393118 5904 ts_manager.cc:194] Re-registered known tserver with Master: 7ed1c497e3454a6eb54f4bcfda8552de (127.31.74.195:41485)
I20260806 02:59:36.393257 7191 heartbeater.cc:511] Master 127.31.74.253:40869 requested a full tablet report, sending...
I20260806 02:59:36.394573 5904 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.74.195@KRBTEST.COM'} at 127.31.74.195:60921
I20260806 02:59:36.394822 6517 ts_manager.cc:194] Registered new tserver with Master: 7ed1c497e3454a6eb54f4bcfda8552de (127.31.74.195:41485)
I20260806 02:59:36.403532 7200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:36.338259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.252:35959 (local address 127.31.74.195:57111)
0806 02:59:36.358656 (+ 20397us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:36.358679 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:36.358846 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:36.359510 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:36.359515 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:36.360240 (+ 725us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:59:36.360744 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:36.360756 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:36.362677 (+ 1921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:36.362685 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:36.364837 (+ 2152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:36.364858 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:36.378119 (+ 13261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:36.378127 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:36.378396 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:36.380618 (+ 2222us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:59:36.380659 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:59:36.389248 (+ 8589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:59:36.399156 (+ 9908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:36.399163 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:36.399166 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:36.399790 (+ 624us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:36.400343 (+ 553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:36.400349 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:36.400354 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:36.400539 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:36.401725 (+ 1186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:59:36.401731 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:59:36.401937 (+ 206us) client_negotiation.cc:769] Sending connection context
0806 02:59:36.402474 (+ 537us) client_negotiation.cc:240] Negotiation successful
0806 02:59:36.402950 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20215,"spinlock_wait_cycles":9088,"thread_start_us":20058,"threads_started":1}
I20260806 02:59:36.404894 7199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:36.338259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.195:57111 (local address 127.31.74.252:35959)
0806 02:59:36.356997 (+ 18738us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:36.357007 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:36.358985 (+ 1978us) server_negotiation.cc:407] Connection header received
0806 02:59:36.359073 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:36.359079 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:36.359187 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:36.359300 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:36.360994 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:36.362237 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:36.366698 (+ 4461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:36.367684 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:36.367868 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:36.394248 (+ 26380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:36.394256 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:36.394263 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:36.394316 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:36.398946 (+ 4630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:36.399956 (+ 1010us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:36.399964 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:36.399971 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:36.400171 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:36.400698 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:36.400703 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:36.400707 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:36.401479 (+ 772us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:36.401646 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:36.404085 (+ 2439us) server_negotiation.cc:299] Negotiation successful
0806 02:59:36.404371 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18468,"thread_start_us":19705,"threads_started":1}
I20260806 02:59:36.406487 7189 heartbeater.cc:347] Connected to a master server at 127.31.74.252:35959
I20260806 02:59:36.406715 7189 heartbeater.cc:464] Registering TS with master...
I20260806 02:59:36.407297 7189 heartbeater.cc:511] Master 127.31.74.252:35959 requested a full tablet report, sending...
I20260806 02:59:36.410969 6709 ts_manager.cc:194] Registered new tserver with Master: 7ed1c497e3454a6eb54f4bcfda8552de (127.31.74.195:41485)
I20260806 02:59:36.413302 7207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:36.362812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48542 (local address 127.31.74.253:40869)
0806 02:59:36.366676 (+ 3864us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:36.366685 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:36.381746 (+ 15061us) server_negotiation.cc:407] Connection header received
0806 02:59:36.387528 (+ 5782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:36.387537 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:36.387649 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:36.387883 (+ 234us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:36.396367 (+ 8484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:36.397867 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:36.400249 (+ 2382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:36.401059 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:36.401170 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:36.405188 (+ 4018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:36.405195 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:36.405200 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:36.405246 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:36.409035 (+ 3789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:36.409851 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:36.409857 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:36.409861 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:36.410030 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:36.410693 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:36.410701 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:36.410710 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:36.411089 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:36.411685 (+ 596us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:36.412329 (+ 644us) server_negotiation.cc:299] Negotiation successful
0806 02:59:36.412570 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3643,"thread_start_us":3650,"threads_started":1}
I20260806 02:59:36.418397 7208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:36.381931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39666 (local address 127.31.74.252:35959)
0806 02:59:36.390276 (+ 8345us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:36.390283 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:36.390638 (+ 355us) server_negotiation.cc:407] Connection header received
0806 02:59:36.390799 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:36.390828 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:36.390935 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:36.391071 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:36.394337 (+ 3266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:36.396692 (+ 2355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:36.399166 (+ 2474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:36.399930 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:36.400068 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:36.404445 (+ 4377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:36.404453 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:36.404458 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:36.404502 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:36.414834 (+ 10332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:36.415645 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:36.415650 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:36.415654 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:36.416035 (+ 381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:36.416571 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:36.416577 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:36.416582 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:36.416935 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:36.417251 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:36.417677 (+ 426us) server_negotiation.cc:299] Negotiation successful
0806 02:59:36.417946 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8172,"thread_start_us":6893,"threads_started":1}
I20260806 02:59:36.421551 7206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:36.362443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46334 (local address 127.31.74.254:36051)
0806 02:59:36.372103 (+ 9660us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:36.372113 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:36.375606 (+ 3493us) server_negotiation.cc:407] Connection header received
0806 02:59:36.375698 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:36.375703 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:36.375818 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:36.375945 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:36.382006 (+ 6061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:36.384722 (+ 2716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:36.392741 (+ 8019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:36.393662 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:36.393782 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:36.401790 (+ 8008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:36.401799 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:36.401804 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:36.401853 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:36.406418 (+ 4565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:36.413622 (+ 7204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:36.413629 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:36.413633 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:36.413875 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:36.419845 (+ 5970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:36.419852 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:36.419855 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:36.420214 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:36.420547 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:36.420883 (+ 336us) server_negotiation.cc:299] Negotiation successful
0806 02:59:36.421129 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9370,"spinlock_wait_cycles":121216,"thread_start_us":9335,"threads_started":1}
I20260806 02:59:36.426237 5904 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46334:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:59:36.436049 5904 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:59:36.487360 7224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:36.462744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36428 (local address 127.31.74.194:41805)
0806 02:59:36.475702 (+ 12958us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:36.475718 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:36.475773 (+ 55us) server_negotiation.cc:407] Connection header received
0806 02:59:36.475927 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:36.475942 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:36.477320 (+ 1378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:36.477694 (+ 374us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:59:36.479175 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:36.481456 (+ 2281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:36.484193 (+ 2737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:36.485723 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:36.485905 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:36.486155 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:36.486792 (+ 637us) server_negotiation.cc:299] Negotiation successful
0806 02:59:36.487044 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12655,"thread_start_us":13839,"threads_started":1}
I20260806 02:59:36.487561 7223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:36.462103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.194:41805 (local address 127.0.0.1:36428)
0806 02:59:36.462817 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:36.462839 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:36.463257 (+ 418us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:36.478074 (+ 14817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:36.478085 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:36.478143 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:59:36.478871 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:36.478890 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:36.481701 (+ 2811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:36.481710 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:36.484023 (+ 2313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:36.484034 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:36.485963 (+ 1929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:36.485970 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:36.486176 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:36.486219 (+ 43us) client_negotiation.cc:769] Sending connection context
0806 02:59:36.486316 (+ 97us) client_negotiation.cc:240] Negotiation successful
0806 02:59:36.486525 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"mutex_wait_us":54,"thread_start_us":859,"threads_started":1}
I20260806 02:59:36.489461 7222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:36.461920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43402 (local address 127.31.74.195:41485)
0806 02:59:36.478129 (+ 16209us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:36.478135 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:36.478163 (+ 28us) server_negotiation.cc:407] Connection header received
0806 02:59:36.478263 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:36.478268 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:36.478495 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:36.478804 (+ 309us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:59:36.479639 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:36.480809 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:36.484869 (+ 4060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:36.487921 (+ 3052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:36.488070 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:36.488323 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:36.488771 (+ 448us) server_negotiation.cc:299] Negotiation successful
0806 02:59:36.488960 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16064,"thread_start_us":17151,"threads_started":1}
I20260806 02:59:36.490224 7221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:36.461564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.195:41485 (local address 127.0.0.1:43402)
0806 02:59:36.462458 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:36.462487 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:36.462726 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:36.478920 (+ 16194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:36.478933 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:36.478966 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:59:36.479423 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:36.479434 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:36.480957 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:36.480965 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:36.483239 (+ 2274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:36.483258 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:36.488158 (+ 4900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:36.488165 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:36.488387 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:36.488423 (+ 36us) client_negotiation.cc:769] Sending connection context
0806 02:59:36.488542 (+ 119us) client_negotiation.cc:240] Negotiation successful
0806 02:59:36.488738 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"mutex_wait_us":27,"thread_start_us":444,"threads_started":1}
I20260806 02:59:36.492406 6979 tablet_service.cc:1467] Processing CreateTablet for tablet 625b133e2fd1449099ce78b04bbbfd85 (DEFAULT_TABLE table=test-table [id=6836b776d7114f7c8aaeae4e6f80223b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:59:36.494467 6979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 625b133e2fd1449099ce78b04bbbfd85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:59:36.494163 7124 tablet_service.cc:1467] Processing CreateTablet for tablet 625b133e2fd1449099ce78b04bbbfd85 (DEFAULT_TABLE table=test-table [id=6836b776d7114f7c8aaeae4e6f80223b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:59:36.495792 7225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:36.466273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:46705 (local address 127.0.0.1:47400)
0806 02:59:36.483268 (+ 16995us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:36.483301 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:36.483461 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:36.484706 (+ 1245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:36.484712 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:36.484743 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:59:36.485280 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:36.485294 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:36.488559 (+ 3265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:36.488567 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:36.492120 (+ 3553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:36.492138 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:36.494825 (+ 2687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:36.494833 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:36.495051 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:36.495101 (+ 50us) client_negotiation.cc:769] Sending connection context
0806 02:59:36.495204 (+ 103us) client_negotiation.cc:240] Negotiation successful
0806 02:59:36.495418 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16781,"thread_start_us":16007,"threads_started":1}
I20260806 02:59:36.495874 7124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 625b133e2fd1449099ce78b04bbbfd85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:59:36.496939 7226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:36.466689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47400 (local address 127.31.74.193:46705)
0806 02:59:36.477715 (+ 11026us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:36.477721 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:36.484172 (+ 6451us) server_negotiation.cc:407] Connection header received
0806 02:59:36.484269 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:36.484275 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:36.484482 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:36.484656 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:59:36.485491 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:36.488406 (+ 2915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:36.492329 (+ 3923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:36.494370 (+ 2041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:36.496020 (+ 1650us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:36.496255 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:36.496352 (+ 97us) server_negotiation.cc:299] Negotiation successful
0806 02:59:36.496572 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10837,"thread_start_us":10679,"threads_started":1}
I20260806 02:59:36.510044 7227 tablet_bootstrap.cc:492] T 625b133e2fd1449099ce78b04bbbfd85 P d45fe0ed8f6b4ddb9b3f31f14ba18ace: Bootstrap starting.
I20260806 02:59:36.511324 6825 tablet_service.cc:1467] Processing CreateTablet for tablet 625b133e2fd1449099ce78b04bbbfd85 (DEFAULT_TABLE table=test-table [id=6836b776d7114f7c8aaeae4e6f80223b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:59:36.512351 7229 tablet_bootstrap.cc:492] T 625b133e2fd1449099ce78b04bbbfd85 P 7ed1c497e3454a6eb54f4bcfda8552de: Bootstrap starting.
I20260806 02:59:36.513042 6825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 625b133e2fd1449099ce78b04bbbfd85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:59:36.514377 7227 tablet_bootstrap.cc:654] T 625b133e2fd1449099ce78b04bbbfd85 P d45fe0ed8f6b4ddb9b3f31f14ba18ace: Neither blocks nor log segments found. Creating new log.
I20260806 02:59:36.515847 7227 log.cc:824] T 625b133e2fd1449099ce78b04bbbfd85 P d45fe0ed8f6b4ddb9b3f31f14ba18ace: Log is configured to *not* fsync() on all Append() calls
I20260806 02:59:36.516489 7229 tablet_bootstrap.cc:654] T 625b133e2fd1449099ce78b04bbbfd85 P 7ed1c497e3454a6eb54f4bcfda8552de: Neither blocks nor log segments found. Creating new log.
I20260806 02:59:36.517891 7229 log.cc:824] T 625b133e2fd1449099ce78b04bbbfd85 P 7ed1c497e3454a6eb54f4bcfda8552de: Log is configured to *not* fsync() on all Append() calls
I20260806 02:59:36.525674 7229 tablet_bootstrap.cc:492] T 625b133e2fd1449099ce78b04bbbfd85 P 7ed1c497e3454a6eb54f4bcfda8552de: No bootstrap required, opened a new log
I20260806 02:59:36.525838 7227 tablet_bootstrap.cc:492] T 625b133e2fd1449099ce78b04bbbfd85 P d45fe0ed8f6b4ddb9b3f31f14ba18ace: No bootstrap required, opened a new log
I20260806 02:59:36.526083 7229 ts_tablet_manager.cc:1397] T 625b133e2fd1449099ce78b04bbbfd85 P 7ed1c497e3454a6eb54f4bcfda8552de: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.004s
I20260806 02:59:36.526165 7227 ts_tablet_manager.cc:1397] T 625b133e2fd1449099ce78b04bbbfd85 P d45fe0ed8f6b4ddb9b3f31f14ba18ace: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.007s
I20260806 02:59:36.536168 7232 tablet_bootstrap.cc:492] T 625b133e2fd1449099ce78b04bbbfd85 P 140e7485da14480cb28af227351c865f: Bootstrap starting.
I20260806 02:59:36.536275 7229 raft_consensus.cc:348] T 625b133e2fd1449099ce78b04bbbfd85 P 7ed1c497e3454a6eb54f4bcfda8552de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ed1c497e3454a6eb54f4bcfda8552de" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 41485 } } peers { permanent_uuid: "d45fe0ed8f6b4ddb9b3f31f14ba18ace" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 41805 } } peers { permanent_uuid: "140e7485da14480cb28af227351c865f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 46705 } }
I20260806 02:59:36.536841 7229 raft_consensus.cc:374] T 625b133e2fd1449099ce78b04bbbfd85 P 7ed1c497e3454a6eb54f4bcfda8552de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:59:36.536962 7229 raft_consensus.cc:729] T 625b133e2fd1449099ce78b04bbbfd85 P 7ed1c497e3454a6eb54f4bcfda8552de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ed1c497e3454a6eb54f4bcfda8552de, State: Initialized, Role: FOLLOWER
I20260806 02:59:36.537894 7229 consensus_queue.cc:260] T 625b133e2fd1449099ce78b04bbbfd85 P 7ed1c497e3454a6eb54f4bcfda8552de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ed1c497e3454a6eb54f4bcfda8552de" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 41485 } } peers { permanent_uuid: "d45fe0ed8f6b4ddb9b3f31f14ba18ace" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 41805 } } peers { permanent_uuid: "140e7485da14480cb28af227351c865f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 46705 } }
I20260806 02:59:36.540355 7229 ts_tablet_manager.cc:1428] T 625b133e2fd1449099ce78b04bbbfd85 P 7ed1c497e3454a6eb54f4bcfda8552de: Time spent starting tablet: real 0.014s user 0.009s sys 0.006s
I20260806 02:59:36.540851 7190 heartbeater.cc:503] Master 127.31.74.254:36051 was elected leader, sending a full tablet report...
I20260806 02:59:36.540668 7227 raft_consensus.cc:348] T 625b133e2fd1449099ce78b04bbbfd85 P d45fe0ed8f6b4ddb9b3f31f14ba18ace [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ed1c497e3454a6eb54f4bcfda8552de" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 41485 } } peers { permanent_uuid: "d45fe0ed8f6b4ddb9b3f31f14ba18ace" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 41805 } } peers { permanent_uuid: "140e7485da14480cb28af227351c865f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 46705 } }
I20260806 02:59:36.541487 7227 raft_consensus.cc:374] T 625b133e2fd1449099ce78b04bbbfd85 P d45fe0ed8f6b4ddb9b3f31f14ba18ace [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:59:36.541625 7227 raft_consensus.cc:729] T 625b133e2fd1449099ce78b04bbbfd85 P d45fe0ed8f6b4ddb9b3f31f14ba18ace [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d45fe0ed8f6b4ddb9b3f31f14ba18ace, State: Initialized, Role: FOLLOWER
I20260806 02:59:36.541782 7232 tablet_bootstrap.cc:654] T 625b133e2fd1449099ce78b04bbbfd85 P 140e7485da14480cb28af227351c865f: Neither blocks nor log segments found. Creating new log.
I20260806 02:59:36.543115 7227 consensus_queue.cc:260] T 625b133e2fd1449099ce78b04bbbfd85 P d45fe0ed8f6b4ddb9b3f31f14ba18ace [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ed1c497e3454a6eb54f4bcfda8552de" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 41485 } } peers { permanent_uuid: "d45fe0ed8f6b4ddb9b3f31f14ba18ace" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 41805 } } peers { permanent_uuid: "140e7485da14480cb28af227351c865f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 46705 } }
I20260806 02:59:36.544219 7232 log.cc:824] T 625b133e2fd1449099ce78b04bbbfd85 P 140e7485da14480cb28af227351c865f: Log is configured to *not* fsync() on all Append() calls
I20260806 02:59:36.547534 7046 heartbeater.cc:503] Master 127.31.74.254:36051 was elected leader, sending a full tablet report...
I20260806 02:59:36.548658 7227 ts_tablet_manager.cc:1428] T 625b133e2fd1449099ce78b04bbbfd85 P d45fe0ed8f6b4ddb9b3f31f14ba18ace: Time spent starting tablet: real 0.022s user 0.009s sys 0.013s
I20260806 02:59:36.551326 7232 tablet_bootstrap.cc:492] T 625b133e2fd1449099ce78b04bbbfd85 P 140e7485da14480cb28af227351c865f: No bootstrap required, opened a new log
I20260806 02:59:36.551687 7232 ts_tablet_manager.cc:1397] T 625b133e2fd1449099ce78b04bbbfd85 P 140e7485da14480cb28af227351c865f: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.005s
I20260806 02:59:36.562776 7232 raft_consensus.cc:348] T 625b133e2fd1449099ce78b04bbbfd85 P 140e7485da14480cb28af227351c865f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ed1c497e3454a6eb54f4bcfda8552de" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 41485 } } peers { permanent_uuid: "d45fe0ed8f6b4ddb9b3f31f14ba18ace" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 41805 } } peers { permanent_uuid: "140e7485da14480cb28af227351c865f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 46705 } }
I20260806 02:59:36.563190 7232 raft_consensus.cc:374] T 625b133e2fd1449099ce78b04bbbfd85 P 140e7485da14480cb28af227351c865f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:59:36.563265 7232 raft_consensus.cc:729] T 625b133e2fd1449099ce78b04bbbfd85 P 140e7485da14480cb28af227351c865f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 140e7485da14480cb28af227351c865f, State: Initialized, Role: FOLLOWER
I20260806 02:59:36.563879 7232 consensus_queue.cc:260] T 625b133e2fd1449099ce78b04bbbfd85 P 140e7485da14480cb28af227351c865f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ed1c497e3454a6eb54f4bcfda8552de" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 41485 } } peers { permanent_uuid: "d45fe0ed8f6b4ddb9b3f31f14ba18ace" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 41805 } } peers { permanent_uuid: "140e7485da14480cb28af227351c865f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 46705 } }
I20260806 02:59:36.565786 7232 ts_tablet_manager.cc:1428] T 625b133e2fd1449099ce78b04bbbfd85 P 140e7485da14480cb28af227351c865f: Time spent starting tablet: real 0.014s user 0.009s sys 0.003s
W20260806 02:59:36.572991 7192 tablet.cc:2369] T 625b133e2fd1449099ce78b04bbbfd85 P 7ed1c497e3454a6eb54f4bcfda8552de: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:59:36.610186 6897 tablet.cc:2369] T 625b133e2fd1449099ce78b04bbbfd85 P 140e7485da14480cb28af227351c865f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:59:36.622275 7048 tablet.cc:2369] T 625b133e2fd1449099ce78b04bbbfd85 P d45fe0ed8f6b4ddb9b3f31f14ba18ace: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:59:36.697337 7234 raft_consensus.cc:482] T 625b133e2fd1449099ce78b04bbbfd85 P d45fe0ed8f6b4ddb9b3f31f14ba18ace [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:59:36.697733 7234 raft_consensus.cc:504] T 625b133e2fd1449099ce78b04bbbfd85 P d45fe0ed8f6b4ddb9b3f31f14ba18ace [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ed1c497e3454a6eb54f4bcfda8552de" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 41485 } } peers { permanent_uuid: "d45fe0ed8f6b4ddb9b3f31f14ba18ace" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 41805 } } peers { permanent_uuid: "140e7485da14480cb28af227351c865f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 46705 } }
I20260806 02:59:36.699759 7234 leader_election.cc:288] T 625b133e2fd1449099ce78b04bbbfd85 P d45fe0ed8f6b4ddb9b3f31f14ba18ace [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7ed1c497e3454a6eb54f4bcfda8552de (127.31.74.195:41485), 140e7485da14480cb28af227351c865f (127.31.74.193:46705)
I20260806 02:59:36.708601 7222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:36.700533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.194:37509 (local address 127.31.74.195:41485)
0806 02:59:36.700884 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:36.700891 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:36.701239 (+ 348us) server_negotiation.cc:407] Connection header received
0806 02:59:36.701584 (+ 345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:36.701595 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:36.701685 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:36.701745 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:59:36.702724 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:36.703702 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:36.706632 (+ 2930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:36.707527 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:36.707741 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:36.707833 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:36.708164 (+ 331us) server_negotiation.cc:299] Negotiation successful
0806 02:59:36.708308 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260806 02:59:36.708936 7237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:36.700174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.195:41485 (local address 127.31.74.194:37509)
0806 02:59:36.700993 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:36.701010 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:36.701310 (+ 300us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:36.701969 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:36.701974 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:36.702006 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:59:36.702522 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:36.702535 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:36.703897 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:36.703909 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:36.706407 (+ 2498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:36.706422 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:36.707787 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:36.707794 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:36.707956 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:36.707982 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 02:59:36.708102 (+ 120us) client_negotiation.cc:240] Negotiation successful
0806 02:59:36.708251 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"mutex_wait_us":63,"thread_start_us":414,"threads_started":1}
I20260806 02:59:36.709919 7226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:36.701301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.194:33743 (local address 127.31.74.193:46705)
0806 02:59:36.701563 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:36.701570 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:36.701980 (+ 410us) server_negotiation.cc:407] Connection header received
0806 02:59:36.702103 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:36.702109 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:36.702199 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:36.702307 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:59:36.703123 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:36.704142 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:36.707369 (+ 3227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:36.708404 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:36.708574 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:36.708667 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:36.709476 (+ 809us) server_negotiation.cc:299] Negotiation successful
0806 02:59:36.709626 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260806 02:59:36.710013 7238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:36.701022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:46705 (local address 127.31.74.194:33743)
0806 02:59:36.701839 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:36.701854 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:36.701985 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:36.702358 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:36.702362 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:36.702386 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:59:36.702917 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:36.702928 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:36.704334 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:36.704350 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:36.707147 (+ 2797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:36.707161 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:36.709035 (+ 1874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:36.709050 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:36.709294 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:36.709333 (+ 39us) client_negotiation.cc:769] Sending connection context
0806 02:59:36.709444 (+ 111us) client_negotiation.cc:240] Negotiation successful
0806 02:59:36.709586 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":709,"thread_start_us":549,"threads_started":1}
I20260806 02:59:36.710132 7144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625b133e2fd1449099ce78b04bbbfd85" candidate_uuid: "d45fe0ed8f6b4ddb9b3f31f14ba18ace" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ed1c497e3454a6eb54f4bcfda8552de" is_pre_election: true
I20260806 02:59:36.711009 7144 raft_consensus.cc:2454] T 625b133e2fd1449099ce78b04bbbfd85 P 7ed1c497e3454a6eb54f4bcfda8552de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d45fe0ed8f6b4ddb9b3f31f14ba18ace in term 0.
I20260806 02:59:36.711184 6845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625b133e2fd1449099ce78b04bbbfd85" candidate_uuid: "d45fe0ed8f6b4ddb9b3f31f14ba18ace" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "140e7485da14480cb28af227351c865f" is_pre_election: true
I20260806 02:59:36.711874 6845 raft_consensus.cc:2454] T 625b133e2fd1449099ce78b04bbbfd85 P 140e7485da14480cb28af227351c865f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d45fe0ed8f6b4ddb9b3f31f14ba18ace in term 0.
I20260806 02:59:36.711899 6928 leader_election.cc:302] T 625b133e2fd1449099ce78b04bbbfd85 P d45fe0ed8f6b4ddb9b3f31f14ba18ace [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ed1c497e3454a6eb54f4bcfda8552de, d45fe0ed8f6b4ddb9b3f31f14ba18ace; no voters:
I20260806 02:59:36.712416 7234 raft_consensus.cc:2781] T 625b133e2fd1449099ce78b04bbbfd85 P d45fe0ed8f6b4ddb9b3f31f14ba18ace [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:59:36.712631 7234 raft_consensus.cc:482] T 625b133e2fd1449099ce78b04bbbfd85 P d45fe0ed8f6b4ddb9b3f31f14ba18ace [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:59:36.712805 7234 raft_consensus.cc:3037] T 625b133e2fd1449099ce78b04bbbfd85 P d45fe0ed8f6b4ddb9b3f31f14ba18ace [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:59:36.715644 7234 raft_consensus.cc:504] T 625b133e2fd1449099ce78b04bbbfd85 P d45fe0ed8f6b4ddb9b3f31f14ba18ace [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ed1c497e3454a6eb54f4bcfda8552de" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 41485 } } peers { permanent_uuid: "d45fe0ed8f6b4ddb9b3f31f14ba18ace" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 41805 } } peers { permanent_uuid: "140e7485da14480cb28af227351c865f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 46705 } }
I20260806 02:59:36.716341 7234 leader_election.cc:288] T 625b133e2fd1449099ce78b04bbbfd85 P d45fe0ed8f6b4ddb9b3f31f14ba18ace [CANDIDATE]: Term 1 election: Requested vote from peers 7ed1c497e3454a6eb54f4bcfda8552de (127.31.74.195:41485), 140e7485da14480cb28af227351c865f (127.31.74.193:46705)
I20260806 02:59:36.716900 7144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625b133e2fd1449099ce78b04bbbfd85" candidate_uuid: "d45fe0ed8f6b4ddb9b3f31f14ba18ace" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ed1c497e3454a6eb54f4bcfda8552de"
I20260806 02:59:36.716989 6845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625b133e2fd1449099ce78b04bbbfd85" candidate_uuid: "d45fe0ed8f6b4ddb9b3f31f14ba18ace" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "140e7485da14480cb28af227351c865f"
I20260806 02:59:36.717139 7144 raft_consensus.cc:3037] T 625b133e2fd1449099ce78b04bbbfd85 P 7ed1c497e3454a6eb54f4bcfda8552de [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:59:36.717322 6845 raft_consensus.cc:3037] T 625b133e2fd1449099ce78b04bbbfd85 P 140e7485da14480cb28af227351c865f [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:59:36.720386 7144 raft_consensus.cc:2454] T 625b133e2fd1449099ce78b04bbbfd85 P 7ed1c497e3454a6eb54f4bcfda8552de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d45fe0ed8f6b4ddb9b3f31f14ba18ace in term 1.
I20260806 02:59:36.720518 6845 raft_consensus.cc:2454] T 625b133e2fd1449099ce78b04bbbfd85 P 140e7485da14480cb28af227351c865f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d45fe0ed8f6b4ddb9b3f31f14ba18ace in term 1.
I20260806 02:59:36.720903 6928 leader_election.cc:302] T 625b133e2fd1449099ce78b04bbbfd85 P d45fe0ed8f6b4ddb9b3f31f14ba18ace [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ed1c497e3454a6eb54f4bcfda8552de, d45fe0ed8f6b4ddb9b3f31f14ba18ace; no voters:
I20260806 02:59:36.721385 7234 raft_consensus.cc:2781] T 625b133e2fd1449099ce78b04bbbfd85 P d45fe0ed8f6b4ddb9b3f31f14ba18ace [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:59:36.721997 7234 raft_consensus.cc:686] T 625b133e2fd1449099ce78b04bbbfd85 P d45fe0ed8f6b4ddb9b3f31f14ba18ace [term 1 LEADER]: Becoming Leader. State: Replica: d45fe0ed8f6b4ddb9b3f31f14ba18ace, State: Running, Role: LEADER
I20260806 02:59:36.722656 7234 consensus_queue.cc:237] T 625b133e2fd1449099ce78b04bbbfd85 P d45fe0ed8f6b4ddb9b3f31f14ba18ace [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ed1c497e3454a6eb54f4bcfda8552de" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 41485 } } peers { permanent_uuid: "d45fe0ed8f6b4ddb9b3f31f14ba18ace" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 41805 } } peers { permanent_uuid: "140e7485da14480cb28af227351c865f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 46705 } }
I20260806 02:59:36.728995 5903 catalog_manager.cc:5495] T 625b133e2fd1449099ce78b04bbbfd85 P d45fe0ed8f6b4ddb9b3f31f14ba18ace reported cstate change: term changed from 0 to 1, leader changed from <none> to d45fe0ed8f6b4ddb9b3f31f14ba18ace (127.31.74.194). New cstate: current_term: 1 leader_uuid: "d45fe0ed8f6b4ddb9b3f31f14ba18ace" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ed1c497e3454a6eb54f4bcfda8552de" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 41485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d45fe0ed8f6b4ddb9b3f31f14ba18ace" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 41805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "140e7485da14480cb28af227351c865f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 46705 } health_report { overall_health: UNKNOWN } } }
I20260806 02:59:36.792423 7224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:36.786066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36438 (local address 127.31.74.194:41805)
0806 02:59:36.786286 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:36.786291 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:36.786310 (+ 19us) server_negotiation.cc:407] Connection header received
0806 02:59:36.786424 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:36.786429 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:36.786537 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:36.786804 (+ 267us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:59:36.787215 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:36.788068 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:36.789682 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:36.790266 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:36.790409 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:36.790984 (+ 575us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:59:36.791714 (+ 730us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:59:36.791882 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:36.792048 (+ 166us) server_negotiation.cc:299] Negotiation successful
0806 02:59:36.792154 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 02:59:36.801808 7144 raft_consensus.cc:1261] T 625b133e2fd1449099ce78b04bbbfd85 P 7ed1c497e3454a6eb54f4bcfda8552de [term 1 FOLLOWER]: Refusing update from remote peer d45fe0ed8f6b4ddb9b3f31f14ba18ace: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:59:36.801860 6845 raft_consensus.cc:1261] T 625b133e2fd1449099ce78b04bbbfd85 P 140e7485da14480cb28af227351c865f [term 1 FOLLOWER]: Refusing update from remote peer d45fe0ed8f6b4ddb9b3f31f14ba18ace: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:59:36.803143 7234 consensus_queue.cc:1035] T 625b133e2fd1449099ce78b04bbbfd85 P d45fe0ed8f6b4ddb9b3f31f14ba18ace [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ed1c497e3454a6eb54f4bcfda8552de" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 41485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:59:36.803579 7239 consensus_queue.cc:1035] T 625b133e2fd1449099ce78b04bbbfd85 P d45fe0ed8f6b4ddb9b3f31f14ba18ace [LEADER]: Connected to new peer: Peer: permanent_uuid: "140e7485da14480cb28af227351c865f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 46705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:59:36.818373 7242 mvcc.cc:205] Tried to move back new op lower bound from 7315395284167811072 to 7315395283860295680. Current Snapshot: MvccSnapshot[applied={T|T < 7315395284167811072}]
I20260806 02:59:36.855278 5903 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46334:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:59:36.855607 5903 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46334:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:59:36.863834 5903 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ec7ce3806c9d48a2bbbfacaed8423300: Sending DeleteTablet for 3 replicas of tablet 625b133e2fd1449099ce78b04bbbfd85
I20260806 02:59:36.865427 7124 tablet_service.cc:1514] Processing DeleteTablet for tablet 625b133e2fd1449099ce78b04bbbfd85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:59:36 UTC) from {username='oryx', principal='oryx/127.31.74.254@KRBTEST.COM'} at 127.0.0.1:43402
I20260806 02:59:36.865645 6979 tablet_service.cc:1514] Processing DeleteTablet for tablet 625b133e2fd1449099ce78b04bbbfd85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:59:36 UTC) from {username='oryx', principal='oryx/127.31.74.254@KRBTEST.COM'} at 127.0.0.1:36428
I20260806 02:59:36.866652 7252 tablet_replica.cc:330] T 625b133e2fd1449099ce78b04bbbfd85 P 7ed1c497e3454a6eb54f4bcfda8552de: stopping tablet replica
I20260806 02:59:36.867141 6825 tablet_service.cc:1514] Processing DeleteTablet for tablet 625b133e2fd1449099ce78b04bbbfd85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:59:36 UTC) from {username='oryx', principal='oryx/127.31.74.254@KRBTEST.COM'} at 127.0.0.1:47400
I20260806 02:59:36.867462 7252 raft_consensus.cc:2229] T 625b133e2fd1449099ce78b04bbbfd85 P 7ed1c497e3454a6eb54f4bcfda8552de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:59:36.868093 7252 raft_consensus.cc:2258] T 625b133e2fd1449099ce78b04bbbfd85 P 7ed1c497e3454a6eb54f4bcfda8552de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:59:36.870843 7252 ts_tablet_manager.cc:1905] T 625b133e2fd1449099ce78b04bbbfd85 P 7ed1c497e3454a6eb54f4bcfda8552de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:59:36.880630 7253 tablet_replica.cc:330] T 625b133e2fd1449099ce78b04bbbfd85 P d45fe0ed8f6b4ddb9b3f31f14ba18ace: stopping tablet replica
I20260806 02:59:36.881319 7253 raft_consensus.cc:2229] T 625b133e2fd1449099ce78b04bbbfd85 P d45fe0ed8f6b4ddb9b3f31f14ba18ace [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:59:36.881809 7254 tablet_replica.cc:330] T 625b133e2fd1449099ce78b04bbbfd85 P 140e7485da14480cb28af227351c865f: stopping tablet replica
I20260806 02:59:36.882258 7254 raft_consensus.cc:2229] T 625b133e2fd1449099ce78b04bbbfd85 P 140e7485da14480cb28af227351c865f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:59:36.882413 7253 raft_consensus.cc:2258] T 625b133e2fd1449099ce78b04bbbfd85 P d45fe0ed8f6b4ddb9b3f31f14ba18ace [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:59:36.882860 7252 ts_tablet_manager.cc:1918] T 625b133e2fd1449099ce78b04bbbfd85 P 7ed1c497e3454a6eb54f4bcfda8552de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:59:36.882942 7254 raft_consensus.cc:2258] T 625b133e2fd1449099ce78b04bbbfd85 P 140e7485da14480cb28af227351c865f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:59:36.883185 7252 log.cc:1196] T 625b133e2fd1449099ce78b04bbbfd85 P 7ed1c497e3454a6eb54f4bcfda8552de: Deleting WAL directory at /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-2/wal/wals/625b133e2fd1449099ce78b04bbbfd85
I20260806 02:59:36.884681 7252 ts_tablet_manager.cc:1939] T 625b133e2fd1449099ce78b04bbbfd85 P 7ed1c497e3454a6eb54f4bcfda8552de: Deleting consensus metadata
I20260806 02:59:36.885128 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 5868
I20260806 02:59:36.886409 5934 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:59:36.890409 7253 ts_tablet_manager.cc:1905] T 625b133e2fd1449099ce78b04bbbfd85 P d45fe0ed8f6b4ddb9b3f31f14ba18ace: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:59:36.894382 7254 ts_tablet_manager.cc:1905] T 625b133e2fd1449099ce78b04bbbfd85 P 140e7485da14480cb28af227351c865f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:59:36.903038 7254 ts_tablet_manager.cc:1918] T 625b133e2fd1449099ce78b04bbbfd85 P 140e7485da14480cb28af227351c865f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:59:36.903301 7254 log.cc:1196] T 625b133e2fd1449099ce78b04bbbfd85 P 140e7485da14480cb28af227351c865f: Deleting WAL directory at /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-0/wal/wals/625b133e2fd1449099ce78b04bbbfd85
I20260806 02:59:36.904048 7254 ts_tablet_manager.cc:1939] T 625b133e2fd1449099ce78b04bbbfd85 P 140e7485da14480cb28af227351c865f: Deleting consensus metadata
I20260806 02:59:36.907097 7253 ts_tablet_manager.cc:1918] T 625b133e2fd1449099ce78b04bbbfd85 P d45fe0ed8f6b4ddb9b3f31f14ba18ace: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:59:36.907405 7253 log.cc:1196] T 625b133e2fd1449099ce78b04bbbfd85 P d45fe0ed8f6b4ddb9b3f31f14ba18ace: Deleting WAL directory at /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-1/wal/wals/625b133e2fd1449099ce78b04bbbfd85
I20260806 02:59:36.908143 7253 ts_tablet_manager.cc:1939] T 625b133e2fd1449099ce78b04bbbfd85 P d45fe0ed8f6b4ddb9b3f31f14ba18ace: Deleting consensus metadata
I20260806 02:59:36.924332 7207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:36.870015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48546 (local address 127.31.74.253:40869)
0806 02:59:36.870227 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:36.870233 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:36.884471 (+ 14238us) server_negotiation.cc:407] Connection header received
0806 02:59:36.885691 (+ 1220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:36.885707 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:36.885803 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:36.885922 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:36.900795 (+ 14873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:36.902007 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:36.908596 (+ 6589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:36.909335 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:36.909559 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:36.914969 (+ 5410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:36.914978 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:36.914983 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:36.915054 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:36.919654 (+ 4600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:36.920914 (+ 1260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:36.920921 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:36.920925 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:36.921116 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:36.921631 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:36.921636 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:36.921640 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:36.922500 (+ 860us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:36.922860 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:36.923672 (+ 812us) server_negotiation.cc:299] Negotiation successful
0806 02:59:36.923956 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 02:59:36.947781 7208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:36.883983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39676 (local address 127.31.74.252:35959)
0806 02:59:36.884231 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:36.884237 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:36.925113 (+ 40876us) server_negotiation.cc:407] Connection header received
0806 02:59:36.925824 (+ 711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:36.925832 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:36.925950 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:36.926089 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:36.929052 (+ 2963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:36.930964 (+ 1912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:36.933599 (+ 2635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:36.934385 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:36.934531 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:36.938741 (+ 4210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:36.938749 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:36.938755 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:36.938805 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:36.943690 (+ 4885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:36.944824 (+ 1134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:36.944831 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:36.944836 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:36.945022 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:36.945550 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:36.945556 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:36.945560 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:36.946039 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:36.946360 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:36.947084 (+ 724us) server_negotiation.cc:299] Negotiation successful
0806 02:59:36.947378 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260806 02:59:37.040067 5888 catalog_manager.cc:4841] TS d45fe0ed8f6b4ddb9b3f31f14ba18ace (127.31.74.194:41805): tablet 625b133e2fd1449099ce78b04bbbfd85 (table test-table [id=6836b776d7114f7c8aaeae4e6f80223b]) successfully deleted
I20260806 02:59:37.040827 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 5868
W20260806 02:59:37.069231 6778 connection.cc:535] server connection from 127.0.0.1:47400 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260806 02:59:37.102048 7208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:37.082819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39678 (local address 127.31.74.252:35959)
0806 02:59:37.083447 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:37.083462 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:37.083736 (+ 274us) server_negotiation.cc:407] Connection header received
0806 02:59:37.083901 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:37.083907 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:37.084023 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:37.084153 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:37.085978 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:37.086806 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:37.090277 (+ 3471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:37.091092 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:37.091270 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:37.095293 (+ 4023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:37.095299 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:37.095302 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:37.095381 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:37.098721 (+ 3340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:37.099671 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:37.099679 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:37.099685 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:37.099878 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:37.100404 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:37.100408 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:37.100410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:37.100716 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:37.100994 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:37.101407 (+ 413us) server_negotiation.cc:299] Negotiation successful
0806 02:59:37.101646 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 02:59:37.103790 7207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:37.081220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48558 (local address 127.31.74.253:40869)
0806 02:59:37.081475 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:37.081484 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:37.082528 (+ 1044us) server_negotiation.cc:407] Connection header received
0806 02:59:37.083105 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:37.083116 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:37.083183 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:37.083303 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:37.085556 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:37.086898 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:37.089025 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:37.089824 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:37.089995 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:37.094498 (+ 4503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:37.094505 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:37.094510 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:37.094561 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:37.098405 (+ 3844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:37.099555 (+ 1150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:37.099560 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:37.099564 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:37.099722 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:37.100376 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:37.100382 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:37.100385 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:37.100745 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:37.100993 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:37.102355 (+ 1362us) server_negotiation.cc:299] Negotiation successful
0806 02:59:37.103204 (+ 849us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260806 02:59:37.836679 7263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:37.835774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:36051 (local address 127.31.74.193:50851)
0806 02:59:37.836332 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:37.836465 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:36051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":296,"threads_started":1}
W20260806 02:59:37.837391 6892 heartbeater.cc:644] Failed to heartbeat to 127.31.74.254:36051 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.31.74.254:36051: connect: Connection refused (error 111)
I20260806 02:59:37.845477 7264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:37.844609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:36051 (local address 127.31.74.195:59225)
0806 02:59:37.845160 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:37.845300 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:36051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":291,"threads_started":1}
W20260806 02:59:37.845914 7190 heartbeater.cc:644] Failed to heartbeat to 127.31.74.254:36051 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.31.74.254:36051: connect: Connection refused (error 111)
I20260806 02:59:37.848523 7265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:37.847395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:36051 (local address 127.31.74.194:46287)
0806 02:59:37.848148 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:37.848286 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:36051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":646,"thread_start_us":291,"threads_started":1}
W20260806 02:59:37.848930 7046 heartbeater.cc:644] Failed to heartbeat to 127.31.74.254:36051 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.31.74.254:36051: connect: Connection refused (error 111)
I20260806 02:59:38.364367 7266 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2698ead82bb0481bbc3bfee291d34fdc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ec7ce3806c9d48a2bbbfacaed8423300)
I20260806 02:59:38.364663 7267 raft_consensus.cc:482] T 00000000000000000000000000000000 P 001d2d57ebfd4d6aa7d81fde49a50231 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ec7ce3806c9d48a2bbbfacaed8423300)
I20260806 02:59:38.365003 7267 raft_consensus.cc:504] T 00000000000000000000000000000000 P 001d2d57ebfd4d6aa7d81fde49a50231 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ec7ce3806c9d48a2bbbfacaed8423300" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36051 } } peers { permanent_uuid: "001d2d57ebfd4d6aa7d81fde49a50231" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 40869 } attrs { promote: false } } peers { permanent_uuid: "2698ead82bb0481bbc3bfee291d34fdc" member_type: VOTER last_known_addr { host: "127.31.74.252" port: 35959 } attrs { promote: false } }
I20260806 02:59:38.364933 7266 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2698ead82bb0481bbc3bfee291d34fdc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ec7ce3806c9d48a2bbbfacaed8423300" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36051 } } peers { permanent_uuid: "001d2d57ebfd4d6aa7d81fde49a50231" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 40869 } attrs { promote: false } } peers { permanent_uuid: "2698ead82bb0481bbc3bfee291d34fdc" member_type: VOTER last_known_addr { host: "127.31.74.252" port: 35959 } attrs { promote: false } }
I20260806 02:59:38.373098 7266 leader_election.cc:288] T 00000000000000000000000000000000 P 2698ead82bb0481bbc3bfee291d34fdc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ec7ce3806c9d48a2bbbfacaed8423300 (127.31.74.254:36051), 001d2d57ebfd4d6aa7d81fde49a50231 (127.31.74.253:40869)
I20260806 02:59:38.373255 7267 leader_election.cc:288] T 00000000000000000000000000000000 P 001d2d57ebfd4d6aa7d81fde49a50231 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ec7ce3806c9d48a2bbbfacaed8423300 (127.31.74.254:36051), 2698ead82bb0481bbc3bfee291d34fdc (127.31.74.252:35959)
I20260806 02:59:38.379393 7270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:38.373646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:36051 (local address 127.0.0.1:38116)
0806 02:59:38.378719 (+ 5073us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:38.379075 (+ 356us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:36051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4940,"thread_start_us":4965,"threads_started":1}
W20260806 02:59:38.379884 6693 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.74.254:36051: connect: Connection refused (error 111)
I20260806 02:59:38.380132 7268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:38.366988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:36051 (local address 127.0.0.1:38124)
0806 02:59:38.379398 (+ 12410us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:38.379741 (+ 343us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:36051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12192,"thread_start_us":12016,"threads_started":1}
W20260806 02:59:38.380465 6501 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.74.254:36051: connect: Connection refused (error 111)
I20260806 02:59:38.385458 7270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:38.384978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:36051 (local address 127.0.0.1:38134)
0806 02:59:38.385186 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:38.385278 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:36051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260806 02:59:38.385699 7269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:38.384826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:36051 (local address 127.0.0.1:38128)
0806 02:59:38.385182 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:38.385425 (+ 243us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:36051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
W20260806 02:59:38.385711 6693 leader_election.cc:334] T 00000000000000000000000000000000 P 2698ead82bb0481bbc3bfee291d34fdc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ec7ce3806c9d48a2bbbfacaed8423300 (127.31.74.254:36051): Network error: Client connection negotiation failed: client connection to 127.31.74.254:36051: connect: Connection refused (error 111)
W20260806 02:59:38.385939 6501 leader_election.cc:334] T 00000000000000000000000000000000 P 001d2d57ebfd4d6aa7d81fde49a50231 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ec7ce3806c9d48a2bbbfacaed8423300 (127.31.74.254:36051): Network error: Client connection negotiation failed: client connection to 127.31.74.254:36051: connect: Connection refused (error 111)
I20260806 02:59:38.390460 7271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:38.367313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45284 (local address 127.31.74.252:35959)
0806 02:59:38.377607 (+ 10294us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:38.377617 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:38.380528 (+ 2911us) server_negotiation.cc:407] Connection header received
0806 02:59:38.381405 (+ 877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:38.381417 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:38.381552 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:38.381661 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:59:38.382947 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:38.384594 (+ 1647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:38.387312 (+ 2718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:38.388771 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:38.388923 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:38.389107 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:38.389938 (+ 831us) server_negotiation.cc:299] Negotiation successful
0806 02:59:38.390111 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":10084,"thread_start_us":9895,"threads_started":1}
I20260806 02:59:38.390496 7268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:38.367055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.252:35959 (local address 127.0.0.1:45284)
0806 02:59:38.380396 (+ 13341us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:38.380421 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:38.380667 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:38.382195 (+ 1528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:38.382205 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:38.382253 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:59:38.382671 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:38.382691 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:38.384831 (+ 2140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:38.384842 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:38.387153 (+ 2311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:38.387165 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:38.389178 (+ 2013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:38.389186 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:38.389672 (+ 486us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:38.389711 (+ 39us) client_negotiation.cc:769] Sending connection context
0806 02:59:38.389860 (+ 149us) client_negotiation.cc:240] Negotiation successful
0806 02:59:38.390106 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13223,"mutex_wait_us":101,"thread_start_us":16524,"threads_started":1}
I20260806 02:59:38.391575 6720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "001d2d57ebfd4d6aa7d81fde49a50231" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2698ead82bb0481bbc3bfee291d34fdc" is_pre_election: true
I20260806 02:59:38.392057 6720 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2698ead82bb0481bbc3bfee291d34fdc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 001d2d57ebfd4d6aa7d81fde49a50231 in term 1.
I20260806 02:59:38.392840 7273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:38.367058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.253:40869 (local address 127.0.0.1:52808)
0806 02:59:38.377782 (+ 10724us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:38.377818 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:38.378048 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:38.382959 (+ 4911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:38.382965 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:38.383004 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:59:38.384654 (+ 1650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:38.384669 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:38.386801 (+ 2132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:38.386806 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:38.389206 (+ 2400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:38.389219 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:38.391828 (+ 2609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:38.391835 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:38.392144 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:38.392171 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 02:59:38.392312 (+ 141us) client_negotiation.cc:240] Negotiation successful
0806 02:59:38.392504 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10556,"mutex_wait_us":60,"thread_start_us":11291,"threads_started":1}
I20260806 02:59:38.392704 6504 leader_election.cc:302] T 00000000000000000000000000000000 P 001d2d57ebfd4d6aa7d81fde49a50231 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 001d2d57ebfd4d6aa7d81fde49a50231, 2698ead82bb0481bbc3bfee291d34fdc; no voters: ec7ce3806c9d48a2bbbfacaed8423300
I20260806 02:59:38.393113 7272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:38.373038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52808 (local address 127.31.74.253:40869)
0806 02:59:38.379889 (+ 6851us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:38.379899 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:38.379928 (+ 29us) server_negotiation.cc:407] Connection header received
0806 02:59:38.380136 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:38.380149 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:38.382715 (+ 2566us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:38.382863 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:59:38.384889 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:38.386639 (+ 1750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:38.389525 (+ 2886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:38.391483 (+ 1958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:38.391785 (+ 302us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:38.391924 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:38.392389 (+ 465us) server_negotiation.cc:299] Negotiation successful
0806 02:59:38.392569 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":107,"server-negotiator.queue_time_us":6685,"thread_start_us":8011,"threads_started":1}
I20260806 02:59:38.393188 7267 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 001d2d57ebfd4d6aa7d81fde49a50231 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 02:59:38.393312 7267 raft_consensus.cc:482] T 00000000000000000000000000000000 P 001d2d57ebfd4d6aa7d81fde49a50231 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ec7ce3806c9d48a2bbbfacaed8423300)
I20260806 02:59:38.393431 7267 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 001d2d57ebfd4d6aa7d81fde49a50231 [term 1 FOLLOWER]: Advancing to term 2
I20260806 02:59:38.393802 6527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2698ead82bb0481bbc3bfee291d34fdc" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "001d2d57ebfd4d6aa7d81fde49a50231" is_pre_election: true
I20260806 02:59:38.395895 7267 raft_consensus.cc:504] T 00000000000000000000000000000000 P 001d2d57ebfd4d6aa7d81fde49a50231 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ec7ce3806c9d48a2bbbfacaed8423300" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36051 } } peers { permanent_uuid: "001d2d57ebfd4d6aa7d81fde49a50231" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 40869 } attrs { promote: false } } peers { permanent_uuid: "2698ead82bb0481bbc3bfee291d34fdc" member_type: VOTER last_known_addr { host: "127.31.74.252" port: 35959 } attrs { promote: false } }
I20260806 02:59:38.396495 6527 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 001d2d57ebfd4d6aa7d81fde49a50231 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2698ead82bb0481bbc3bfee291d34fdc in current term 2: Already voted for candidate 001d2d57ebfd4d6aa7d81fde49a50231 in this term.
I20260806 02:59:38.397096 7267 leader_election.cc:288] T 00000000000000000000000000000000 P 001d2d57ebfd4d6aa7d81fde49a50231 [CANDIDATE]: Term 2 election: Requested vote from peers ec7ce3806c9d48a2bbbfacaed8423300 (127.31.74.254:36051), 2698ead82bb0481bbc3bfee291d34fdc (127.31.74.252:35959)
I20260806 02:59:38.397372 6720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "001d2d57ebfd4d6aa7d81fde49a50231" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2698ead82bb0481bbc3bfee291d34fdc"
I20260806 02:59:38.397651 6720 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2698ead82bb0481bbc3bfee291d34fdc [term 1 FOLLOWER]: Advancing to term 2
I20260806 02:59:38.397806 7268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:38.397143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:36051 (local address 127.0.0.1:38138)
0806 02:59:38.397500 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:38.397619 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:36051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237}
I20260806 02:59:38.397730 6696 leader_election.cc:302] T 00000000000000000000000000000000 P 2698ead82bb0481bbc3bfee291d34fdc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2698ead82bb0481bbc3bfee291d34fdc; no voters: 001d2d57ebfd4d6aa7d81fde49a50231, ec7ce3806c9d48a2bbbfacaed8423300
I20260806 02:59:38.398940 7268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:38.398566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:36051 (local address 127.0.0.1:38146)
0806 02:59:38.398754 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:38.398828 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:36051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260806 02:59:38.399200 6501 leader_election.cc:334] T 00000000000000000000000000000000 P 001d2d57ebfd4d6aa7d81fde49a50231 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ec7ce3806c9d48a2bbbfacaed8423300 (127.31.74.254:36051): Network error: Client connection negotiation failed: client connection to 127.31.74.254:36051: connect: Connection refused (error 111)
I20260806 02:59:38.400421 6720 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2698ead82bb0481bbc3bfee291d34fdc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 001d2d57ebfd4d6aa7d81fde49a50231 in term 2.
I20260806 02:59:38.400606 7266 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 2698ead82bb0481bbc3bfee291d34fdc [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 02:59:38.401064 6504 leader_election.cc:302] T 00000000000000000000000000000000 P 001d2d57ebfd4d6aa7d81fde49a50231 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 001d2d57ebfd4d6aa7d81fde49a50231, 2698ead82bb0481bbc3bfee291d34fdc; no voters: ec7ce3806c9d48a2bbbfacaed8423300
I20260806 02:59:38.401580 7267 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 001d2d57ebfd4d6aa7d81fde49a50231 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 02:59:38.402196 7267 raft_consensus.cc:686] T 00000000000000000000000000000000 P 001d2d57ebfd4d6aa7d81fde49a50231 [term 2 LEADER]: Becoming Leader. State: Replica: 001d2d57ebfd4d6aa7d81fde49a50231, State: Running, Role: LEADER
I20260806 02:59:38.402737 7267 consensus_queue.cc:237] T 00000000000000000000000000000000 P 001d2d57ebfd4d6aa7d81fde49a50231 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ec7ce3806c9d48a2bbbfacaed8423300" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36051 } } peers { permanent_uuid: "001d2d57ebfd4d6aa7d81fde49a50231" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 40869 } attrs { promote: false } } peers { permanent_uuid: "2698ead82bb0481bbc3bfee291d34fdc" member_type: VOTER last_known_addr { host: "127.31.74.252" port: 35959 } attrs { promote: false } }
I20260806 02:59:38.405370 7276 sys_catalog.cc:455] T 00000000000000000000000000000000 P 001d2d57ebfd4d6aa7d81fde49a50231 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 001d2d57ebfd4d6aa7d81fde49a50231. Latest consensus state: current_term: 2 leader_uuid: "001d2d57ebfd4d6aa7d81fde49a50231" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ec7ce3806c9d48a2bbbfacaed8423300" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36051 } } peers { permanent_uuid: "001d2d57ebfd4d6aa7d81fde49a50231" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 40869 } attrs { promote: false } } peers { permanent_uuid: "2698ead82bb0481bbc3bfee291d34fdc" member_type: VOTER last_known_addr { host: "127.31.74.252" port: 35959 } attrs { promote: false } } }
I20260806 02:59:38.405877 7276 sys_catalog.cc:458] T 00000000000000000000000000000000 P 001d2d57ebfd4d6aa7d81fde49a50231 [sys.catalog]: This master's current role is: LEADER
I20260806 02:59:38.406723 7278 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:59:38.412340 7278 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:59:38.413616 7278 catalog_manager.cc:1259] Loaded cluster ID: 8e6a631af4aa433eb774f4779a94b441
I20260806 02:59:38.413717 7278 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:59:38.415671 7278 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:59:38.416832 7278 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 001d2d57ebfd4d6aa7d81fde49a50231: Loaded TSK: 0
I20260806 02:59:38.418497 7278 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:59:38.465905 6516 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48558:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:59:38.473671 6516 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:59:38.478137 7268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:38.477660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:36051 (local address 127.0.0.1:38180)
0806 02:59:38.477904 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:38.477985 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:36051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260806 02:59:38.478166 6720 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2698ead82bb0481bbc3bfee291d34fdc [term 2 FOLLOWER]: Refusing update from remote peer 001d2d57ebfd4d6aa7d81fde49a50231: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260806 02:59:38.479029 7276 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 001d2d57ebfd4d6aa7d81fde49a50231 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2698ead82bb0481bbc3bfee291d34fdc" member_type: VOTER last_known_addr { host: "127.31.74.252" port: 35959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260806 02:59:38.479275 7268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:38.478920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:36051 (local address 127.0.0.1:38196)
0806 02:59:38.479048 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:38.479113 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:36051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260806 02:59:38.479460 6501 consensus_peers.cc:487] T 00000000000000000000000000000000 P 001d2d57ebfd4d6aa7d81fde49a50231 -> Peer ec7ce3806c9d48a2bbbfacaed8423300 (127.31.74.254:36051): Couldn't send request to peer ec7ce3806c9d48a2bbbfacaed8423300. Status: Network error: Client connection negotiation failed: client connection to 127.31.74.254:36051: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260806 02:59:38.482488 7266 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2698ead82bb0481bbc3bfee291d34fdc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 001d2d57ebfd4d6aa7d81fde49a50231. Latest consensus state: current_term: 2 leader_uuid: "001d2d57ebfd4d6aa7d81fde49a50231" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ec7ce3806c9d48a2bbbfacaed8423300" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36051 } } peers { permanent_uuid: "001d2d57ebfd4d6aa7d81fde49a50231" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 40869 } attrs { promote: false } } peers { permanent_uuid: "2698ead82bb0481bbc3bfee291d34fdc" member_type: VOTER last_known_addr { host: "127.31.74.252" port: 35959 } attrs { promote: false } } }
I20260806 02:59:38.482862 7266 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2698ead82bb0481bbc3bfee291d34fdc [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:59:38.485217 7267 sys_catalog.cc:455] T 00000000000000000000000000000000 P 001d2d57ebfd4d6aa7d81fde49a50231 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "001d2d57ebfd4d6aa7d81fde49a50231" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ec7ce3806c9d48a2bbbfacaed8423300" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36051 } } peers { permanent_uuid: "001d2d57ebfd4d6aa7d81fde49a50231" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 40869 } attrs { promote: false } } peers { permanent_uuid: "2698ead82bb0481bbc3bfee291d34fdc" member_type: VOTER last_known_addr { host: "127.31.74.252" port: 35959 } attrs { promote: false } } }
I20260806 02:59:38.485615 7267 sys_catalog.cc:458] T 00000000000000000000000000000000 P 001d2d57ebfd4d6aa7d81fde49a50231 [sys.catalog]: This master's current role is: LEADER
I20260806 02:59:38.486078 7266 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2698ead82bb0481bbc3bfee291d34fdc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "001d2d57ebfd4d6aa7d81fde49a50231" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ec7ce3806c9d48a2bbbfacaed8423300" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36051 } } peers { permanent_uuid: "001d2d57ebfd4d6aa7d81fde49a50231" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 40869 } attrs { promote: false } } peers { permanent_uuid: "2698ead82bb0481bbc3bfee291d34fdc" member_type: VOTER last_known_addr { host: "127.31.74.252" port: 35959 } attrs { promote: false } } }
I20260806 02:59:38.486364 7266 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2698ead82bb0481bbc3bfee291d34fdc [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:59:38.505518 7269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:38.495089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.195:41485 (local address 127.0.0.1:57762)
0806 02:59:38.495269 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:38.495292 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:38.495427 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:38.496281 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:38.496286 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:38.496310 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:59:38.496695 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:38.496708 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:38.499431 (+ 2723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:38.499440 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:38.502365 (+ 2925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:38.502385 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:38.504242 (+ 1857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:38.504251 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:38.504560 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:38.504617 (+ 57us) client_negotiation.cc:769] Sending connection context
0806 02:59:38.504763 (+ 146us) client_negotiation.cc:240] Negotiation successful
0806 02:59:38.504965 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":3}
I20260806 02:59:38.506464 7268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:38.494756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:46705 (local address 127.0.0.1:55758)
0806 02:59:38.494966 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:38.494989 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:38.495090 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:38.496704 (+ 1614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:38.496711 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:38.496748 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:59:38.496996 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:38.497010 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:38.499574 (+ 2564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:38.499587 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:38.504371 (+ 4784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:38.504386 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:38.505933 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:38.505940 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:38.506087 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:38.506111 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 02:59:38.506164 (+ 53us) client_negotiation.cc:240] Negotiation successful
0806 02:59:38.506253 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":2903}
I20260806 02:59:38.506810 7285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:38.495459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57762 (local address 127.31.74.195:41485)
0806 02:59:38.495936 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:38.495945 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:38.495965 (+ 20us) server_negotiation.cc:407] Connection header received
0806 02:59:38.496039 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:38.496044 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:38.496118 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:38.496859 (+ 741us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:59:38.497043 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:38.499219 (+ 2176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:38.502753 (+ 3534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:38.503957 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:38.504091 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:38.504192 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:38.506100 (+ 1908us) server_negotiation.cc:299] Negotiation successful
0806 02:59:38.506376 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":264,"threads_started":1}
I20260806 02:59:38.507400 7284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:38.494868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55758 (local address 127.31.74.193:46705)
0806 02:59:38.496334 (+ 1466us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:38.496341 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:38.496362 (+ 21us) server_negotiation.cc:407] Connection header received
0806 02:59:38.496436 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:38.496440 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:38.496510 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:38.497179 (+ 669us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:59:38.497381 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:38.499335 (+ 1954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:38.504556 (+ 5221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:38.505732 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:38.506737 (+ 1005us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:38.506872 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:38.506952 (+ 80us) server_negotiation.cc:299] Negotiation successful
0806 02:59:38.507117 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1335,"thread_start_us":1215,"threads_started":1}
I20260806 02:59:38.507701 7124 tablet_service.cc:1467] Processing CreateTablet for tablet a7d896e3a3ba439598e3bf2e8e576bdf (DEFAULT_TABLE table=test-table [id=674823d99f03425ba78981f8b347f4e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:59:38.508210 7124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7d896e3a3ba439598e3bf2e8e576bdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:59:38.508153 6825 tablet_service.cc:1467] Processing CreateTablet for tablet a7d896e3a3ba439598e3bf2e8e576bdf (DEFAULT_TABLE table=test-table [id=674823d99f03425ba78981f8b347f4e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:59:38.508579 6825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7d896e3a3ba439598e3bf2e8e576bdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:59:38.513831 7286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:38.496064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.194:41805 (local address 127.0.0.1:38032)
0806 02:59:38.502634 (+ 6570us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:38.502664 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:38.502768 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:38.508142 (+ 5374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:38.508146 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:38.508173 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:59:38.508478 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:38.508489 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:38.509578 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:38.509586 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:38.511116 (+ 1530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:38.511129 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:38.512903 (+ 1774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:38.512910 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:38.513112 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:38.513153 (+ 41us) client_negotiation.cc:769] Sending connection context
0806 02:59:38.513258 (+ 105us) client_negotiation.cc:240] Negotiation successful
0806 02:59:38.513415 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6414,"thread_start_us":6274,"threads_started":1}
I20260806 02:59:38.514093 7287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:38.496467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38032 (local address 127.31.74.194:41805)
0806 02:59:38.507810 (+ 11343us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:38.507818 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:38.507840 (+ 22us) server_negotiation.cc:407] Connection header received
0806 02:59:38.507922 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:38.507928 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:38.508005 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:38.508122 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:59:38.508687 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:38.509442 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:38.511416 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:38.512658 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:38.512836 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:38.513392 (+ 556us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:38.513449 (+ 57us) server_negotiation.cc:299] Negotiation successful
0806 02:59:38.513604 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11216,"thread_start_us":11334,"threads_started":1}
I20260806 02:59:38.515017 6979 tablet_service.cc:1467] Processing CreateTablet for tablet a7d896e3a3ba439598e3bf2e8e576bdf (DEFAULT_TABLE table=test-table [id=674823d99f03425ba78981f8b347f4e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:59:38.515604 6979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7d896e3a3ba439598e3bf2e8e576bdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:59:38.516192 7288 tablet_bootstrap.cc:492] T a7d896e3a3ba439598e3bf2e8e576bdf P 140e7485da14480cb28af227351c865f: Bootstrap starting.
I20260806 02:59:38.518507 7288 tablet_bootstrap.cc:654] T a7d896e3a3ba439598e3bf2e8e576bdf P 140e7485da14480cb28af227351c865f: Neither blocks nor log segments found. Creating new log.
I20260806 02:59:38.518959 7289 tablet_bootstrap.cc:492] T a7d896e3a3ba439598e3bf2e8e576bdf P 7ed1c497e3454a6eb54f4bcfda8552de: Bootstrap starting.
I20260806 02:59:38.520560 7289 tablet_bootstrap.cc:654] T a7d896e3a3ba439598e3bf2e8e576bdf P 7ed1c497e3454a6eb54f4bcfda8552de: Neither blocks nor log segments found. Creating new log.
I20260806 02:59:38.521486 7288 tablet_bootstrap.cc:492] T a7d896e3a3ba439598e3bf2e8e576bdf P 140e7485da14480cb28af227351c865f: No bootstrap required, opened a new log
I20260806 02:59:38.521678 7288 ts_tablet_manager.cc:1397] T a7d896e3a3ba439598e3bf2e8e576bdf P 140e7485da14480cb28af227351c865f: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260806 02:59:38.522509 7289 tablet_bootstrap.cc:492] T a7d896e3a3ba439598e3bf2e8e576bdf P 7ed1c497e3454a6eb54f4bcfda8552de: No bootstrap required, opened a new log
I20260806 02:59:38.522706 7289 ts_tablet_manager.cc:1397] T a7d896e3a3ba439598e3bf2e8e576bdf P 7ed1c497e3454a6eb54f4bcfda8552de: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260806 02:59:38.522503 7288 raft_consensus.cc:348] T a7d896e3a3ba439598e3bf2e8e576bdf P 140e7485da14480cb28af227351c865f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "140e7485da14480cb28af227351c865f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 46705 } } peers { permanent_uuid: "7ed1c497e3454a6eb54f4bcfda8552de" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 41485 } } peers { permanent_uuid: "d45fe0ed8f6b4ddb9b3f31f14ba18ace" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 41805 } }
I20260806 02:59:38.522904 7288 raft_consensus.cc:374] T a7d896e3a3ba439598e3bf2e8e576bdf P 140e7485da14480cb28af227351c865f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:59:38.523028 7288 raft_consensus.cc:729] T a7d896e3a3ba439598e3bf2e8e576bdf P 140e7485da14480cb28af227351c865f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 140e7485da14480cb28af227351c865f, State: Initialized, Role: FOLLOWER
I20260806 02:59:38.523612 7289 raft_consensus.cc:348] T a7d896e3a3ba439598e3bf2e8e576bdf P 7ed1c497e3454a6eb54f4bcfda8552de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "140e7485da14480cb28af227351c865f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 46705 } } peers { permanent_uuid: "7ed1c497e3454a6eb54f4bcfda8552de" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 41485 } } peers { permanent_uuid: "d45fe0ed8f6b4ddb9b3f31f14ba18ace" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 41805 } }
I20260806 02:59:38.523871 7289 raft_consensus.cc:374] T a7d896e3a3ba439598e3bf2e8e576bdf P 7ed1c497e3454a6eb54f4bcfda8552de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:59:38.523968 7289 raft_consensus.cc:729] T a7d896e3a3ba439598e3bf2e8e576bdf P 7ed1c497e3454a6eb54f4bcfda8552de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ed1c497e3454a6eb54f4bcfda8552de, State: Initialized, Role: FOLLOWER
I20260806 02:59:38.523984 7293 tablet_bootstrap.cc:492] T a7d896e3a3ba439598e3bf2e8e576bdf P d45fe0ed8f6b4ddb9b3f31f14ba18ace: Bootstrap starting.
I20260806 02:59:38.524250 7289 consensus_queue.cc:260] T a7d896e3a3ba439598e3bf2e8e576bdf P 7ed1c497e3454a6eb54f4bcfda8552de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "140e7485da14480cb28af227351c865f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 46705 } } peers { permanent_uuid: "7ed1c497e3454a6eb54f4bcfda8552de" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 41485 } } peers { permanent_uuid: "d45fe0ed8f6b4ddb9b3f31f14ba18ace" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 41805 } }
I20260806 02:59:38.526991 7288 consensus_queue.cc:260] T a7d896e3a3ba439598e3bf2e8e576bdf P 140e7485da14480cb28af227351c865f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "140e7485da14480cb28af227351c865f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 46705 } } peers { permanent_uuid: "7ed1c497e3454a6eb54f4bcfda8552de" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 41485 } } peers { permanent_uuid: "d45fe0ed8f6b4ddb9b3f31f14ba18ace" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 41805 } }
I20260806 02:59:38.527413 7293 tablet_bootstrap.cc:654] T a7d896e3a3ba439598e3bf2e8e576bdf P d45fe0ed8f6b4ddb9b3f31f14ba18ace: Neither blocks nor log segments found. Creating new log.
I20260806 02:59:38.527522 7292 raft_consensus.cc:482] T a7d896e3a3ba439598e3bf2e8e576bdf P 140e7485da14480cb28af227351c865f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:59:38.527786 7292 raft_consensus.cc:504] T a7d896e3a3ba439598e3bf2e8e576bdf P 140e7485da14480cb28af227351c865f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "140e7485da14480cb28af227351c865f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 46705 } } peers { permanent_uuid: "7ed1c497e3454a6eb54f4bcfda8552de" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 41485 } } peers { permanent_uuid: "d45fe0ed8f6b4ddb9b3f31f14ba18ace" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 41805 } }
I20260806 02:59:38.536588 7293 tablet_bootstrap.cc:492] T a7d896e3a3ba439598e3bf2e8e576bdf P d45fe0ed8f6b4ddb9b3f31f14ba18ace: No bootstrap required, opened a new log
I20260806 02:59:38.536834 7293 ts_tablet_manager.cc:1397] T a7d896e3a3ba439598e3bf2e8e576bdf P d45fe0ed8f6b4ddb9b3f31f14ba18ace: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.008s
I20260806 02:59:38.538030 7293 raft_consensus.cc:348] T a7d896e3a3ba439598e3bf2e8e576bdf P d45fe0ed8f6b4ddb9b3f31f14ba18ace [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "140e7485da14480cb28af227351c865f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 46705 } } peers { permanent_uuid: "7ed1c497e3454a6eb54f4bcfda8552de" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 41485 } } peers { permanent_uuid: "d45fe0ed8f6b4ddb9b3f31f14ba18ace" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 41805 } }
I20260806 02:59:38.538902 7293 raft_consensus.cc:374] T a7d896e3a3ba439598e3bf2e8e576bdf P d45fe0ed8f6b4ddb9b3f31f14ba18ace [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:59:38.539028 7293 raft_consensus.cc:729] T a7d896e3a3ba439598e3bf2e8e576bdf P d45fe0ed8f6b4ddb9b3f31f14ba18ace [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d45fe0ed8f6b4ddb9b3f31f14ba18ace, State: Initialized, Role: FOLLOWER
I20260806 02:59:38.539343 7293 consensus_queue.cc:260] T a7d896e3a3ba439598e3bf2e8e576bdf P d45fe0ed8f6b4ddb9b3f31f14ba18ace [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "140e7485da14480cb28af227351c865f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 46705 } } peers { permanent_uuid: "7ed1c497e3454a6eb54f4bcfda8552de" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 41485 } } peers { permanent_uuid: "d45fe0ed8f6b4ddb9b3f31f14ba18ace" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 41805 } }
I20260806 02:59:38.539522 7288 ts_tablet_manager.cc:1428] T a7d896e3a3ba439598e3bf2e8e576bdf P 140e7485da14480cb28af227351c865f: Time spent starting tablet: real 0.018s user 0.003s sys 0.002s
I20260806 02:59:38.539640 7298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:38.528829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:36051 (local address 127.31.74.193:41907)
0806 02:59:38.539104 (+ 10275us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:38.539312 (+ 208us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:36051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":10112,"threads_started":1}
I20260806 02:59:38.542639 6893 heartbeater.cc:503] Master 127.31.74.253:40869 was elected leader, sending a full tablet report...
I20260806 02:59:38.546911 7293 ts_tablet_manager.cc:1428] T a7d896e3a3ba439598e3bf2e8e576bdf P d45fe0ed8f6b4ddb9b3f31f14ba18ace: Time spent starting tablet: real 0.010s user 0.002s sys 0.008s
I20260806 02:59:38.548249 7292 leader_election.cc:288] T a7d896e3a3ba439598e3bf2e8e576bdf P 140e7485da14480cb28af227351c865f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7ed1c497e3454a6eb54f4bcfda8552de (127.31.74.195:41485), d45fe0ed8f6b4ddb9b3f31f14ba18ace (127.31.74.194:41805)
I20260806 02:59:38.552163 7300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:38.550922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:36051 (local address 127.31.74.194:55835)
0806 02:59:38.551579 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:38.551828 (+ 249us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:36051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":366,"threads_started":1}
I20260806 02:59:38.553030 7289 ts_tablet_manager.cc:1428] T a7d896e3a3ba439598e3bf2e8e576bdf P 7ed1c497e3454a6eb54f4bcfda8552de: Time spent starting tablet: real 0.026s user 0.007s sys 0.011s
I20260806 02:59:38.553385 7301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:38.549609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:36051 (local address 127.31.74.195:58121)
0806 02:59:38.552819 (+ 3210us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:38.553020 (+ 201us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:36051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3049,"thread_start_us":3205,"threads_started":1}
I20260806 02:59:38.554028 7047 heartbeater.cc:503] Master 127.31.74.253:40869 was elected leader, sending a full tablet report...
I20260806 02:59:38.554222 7191 heartbeater.cc:503] Master 127.31.74.253:40869 was elected leader, sending a full tablet report...
I20260806 02:59:38.556915 7300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:38.553332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:36051 (local address 127.31.74.194:35599)
0806 02:59:38.556635 (+ 3303us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:38.556748 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:36051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329}
I20260806 02:59:38.560770 7301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:38.554082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:36051 (local address 127.31.74.195:52197)
0806 02:59:38.560448 (+ 6366us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:38.560558 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:36051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":6251,"threads_started":1}
I20260806 02:59:38.561574 7287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:38.538560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.193:34691 (local address 127.31.74.194:41805)
0806 02:59:38.538959 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:38.538968 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:38.540028 (+ 1060us) server_negotiation.cc:407] Connection header received
0806 02:59:38.541995 (+ 1967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:38.542004 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:38.542115 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:38.542199 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:59:38.544134 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:38.546615 (+ 2481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:38.557444 (+ 10829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:38.560006 (+ 2562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:38.560203 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:38.560306 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:38.560989 (+ 683us) server_negotiation.cc:299] Negotiation successful
0806 02:59:38.561199 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260806 02:59:38.563112 7298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:38.538345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.194:41805 (local address 127.31.74.193:34691)
0806 02:59:38.539892 (+ 1547us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:38.539916 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:38.541524 (+ 1608us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:38.542512 (+ 988us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:38.542520 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:38.542610 (+ 90us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:59:38.543039 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:38.543054 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:38.547031 (+ 3977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:38.547038 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:38.557037 (+ 9999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:38.557056 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:38.560521 (+ 3465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:38.560527 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:38.560749 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:38.560796 (+ 47us) client_negotiation.cc:769] Sending connection context
0806 02:59:38.561869 (+ 1073us) client_negotiation.cc:240] Negotiation successful
0806 02:59:38.562121 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1415,"spinlock_wait_cycles":2432,"thread_start_us":405,"threads_started":1}
I20260806 02:59:38.564482 7000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7d896e3a3ba439598e3bf2e8e576bdf" candidate_uuid: "140e7485da14480cb28af227351c865f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d45fe0ed8f6b4ddb9b3f31f14ba18ace" is_pre_election: true
I20260806 02:59:38.565076 7000 raft_consensus.cc:2454] T a7d896e3a3ba439598e3bf2e8e576bdf P d45fe0ed8f6b4ddb9b3f31f14ba18ace [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 140e7485da14480cb28af227351c865f in term 0.
I20260806 02:59:38.565871 6780 leader_election.cc:302] T a7d896e3a3ba439598e3bf2e8e576bdf P 140e7485da14480cb28af227351c865f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 140e7485da14480cb28af227351c865f, d45fe0ed8f6b4ddb9b3f31f14ba18ace; no voters:
I20260806 02:59:38.566380 7292 raft_consensus.cc:2781] T a7d896e3a3ba439598e3bf2e8e576bdf P 140e7485da14480cb28af227351c865f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:59:38.566520 7292 raft_consensus.cc:482] T a7d896e3a3ba439598e3bf2e8e576bdf P 140e7485da14480cb28af227351c865f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:59:38.566676 7292 raft_consensus.cc:3037] T a7d896e3a3ba439598e3bf2e8e576bdf P 140e7485da14480cb28af227351c865f [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:59:38.566857 7298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:38.556358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:36051 (local address 127.31.74.193:49857)
0806 02:59:38.566500 (+ 10142us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:38.566644 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:36051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":10024,"threads_started":1}
I20260806 02:59:38.568956 7285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:38.550657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.193:47611 (local address 127.31.74.195:41485)
0806 02:59:38.551565 (+ 908us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:38.551571 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:38.555997 (+ 4426us) server_negotiation.cc:407] Connection header received
0806 02:59:38.556868 (+ 871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:38.556877 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:38.556969 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:38.557038 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:59:38.558199 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:38.559771 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:38.565748 (+ 5977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:38.567752 (+ 2004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:38.567974 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:38.568064 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:38.568471 (+ 407us) server_negotiation.cc:299] Negotiation successful
0806 02:59:38.568616 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":733}
I20260806 02:59:38.568985 7297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:38.555274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.195:41485 (local address 127.31.74.193:47611)
0806 02:59:38.555883 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:38.555905 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:38.556664 (+ 759us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:38.557410 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:38.557417 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:38.557449 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:59:38.558010 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:38.558025 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:38.561992 (+ 3967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:38.562003 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:38.565382 (+ 3379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:38.565412 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:38.568028 (+ 2616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:38.568040 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:38.568264 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:38.568295 (+ 31us) client_negotiation.cc:769] Sending connection context
0806 02:59:38.568437 (+ 142us) client_negotiation.cc:240] Negotiation successful
0806 02:59:38.568603 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429}
I20260806 02:59:38.569460 7292 raft_consensus.cc:504] T a7d896e3a3ba439598e3bf2e8e576bdf P 140e7485da14480cb28af227351c865f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "140e7485da14480cb28af227351c865f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 46705 } } peers { permanent_uuid: "7ed1c497e3454a6eb54f4bcfda8552de" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 41485 } } peers { permanent_uuid: "d45fe0ed8f6b4ddb9b3f31f14ba18ace" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 41805 } }
I20260806 02:59:38.569686 7144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7d896e3a3ba439598e3bf2e8e576bdf" candidate_uuid: "140e7485da14480cb28af227351c865f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ed1c497e3454a6eb54f4bcfda8552de" is_pre_election: true
I20260806 02:59:38.569936 7144 raft_consensus.cc:2454] T a7d896e3a3ba439598e3bf2e8e576bdf P 7ed1c497e3454a6eb54f4bcfda8552de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 140e7485da14480cb28af227351c865f in term 0.
I20260806 02:59:38.570328 7292 leader_election.cc:288] T a7d896e3a3ba439598e3bf2e8e576bdf P 140e7485da14480cb28af227351c865f [CANDIDATE]: Term 1 election: Requested vote from peers 7ed1c497e3454a6eb54f4bcfda8552de (127.31.74.195:41485), d45fe0ed8f6b4ddb9b3f31f14ba18ace (127.31.74.194:41805)
I20260806 02:59:38.570933 7144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7d896e3a3ba439598e3bf2e8e576bdf" candidate_uuid: "140e7485da14480cb28af227351c865f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ed1c497e3454a6eb54f4bcfda8552de"
I20260806 02:59:38.570938 7000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7d896e3a3ba439598e3bf2e8e576bdf" candidate_uuid: "140e7485da14480cb28af227351c865f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d45fe0ed8f6b4ddb9b3f31f14ba18ace"
I20260806 02:59:38.571152 7000 raft_consensus.cc:3037] T a7d896e3a3ba439598e3bf2e8e576bdf P d45fe0ed8f6b4ddb9b3f31f14ba18ace [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:59:38.571187 7144 raft_consensus.cc:3037] T a7d896e3a3ba439598e3bf2e8e576bdf P 7ed1c497e3454a6eb54f4bcfda8552de [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:59:38.574652 7144 raft_consensus.cc:2454] T a7d896e3a3ba439598e3bf2e8e576bdf P 7ed1c497e3454a6eb54f4bcfda8552de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 140e7485da14480cb28af227351c865f in term 1.
I20260806 02:59:38.574652 7000 raft_consensus.cc:2454] T a7d896e3a3ba439598e3bf2e8e576bdf P d45fe0ed8f6b4ddb9b3f31f14ba18ace [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 140e7485da14480cb28af227351c865f in term 1.
I20260806 02:59:38.575135 6780 leader_election.cc:302] T a7d896e3a3ba439598e3bf2e8e576bdf P 140e7485da14480cb28af227351c865f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 140e7485da14480cb28af227351c865f, d45fe0ed8f6b4ddb9b3f31f14ba18ace; no voters:
I20260806 02:59:38.575501 7292 raft_consensus.cc:2781] T a7d896e3a3ba439598e3bf2e8e576bdf P 140e7485da14480cb28af227351c865f [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:59:38.575685 7292 raft_consensus.cc:686] T a7d896e3a3ba439598e3bf2e8e576bdf P 140e7485da14480cb28af227351c865f [term 1 LEADER]: Becoming Leader. State: Replica: 140e7485da14480cb28af227351c865f, State: Running, Role: LEADER
I20260806 02:59:38.576341 7292 consensus_queue.cc:237] T a7d896e3a3ba439598e3bf2e8e576bdf P 140e7485da14480cb28af227351c865f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "140e7485da14480cb28af227351c865f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 46705 } } peers { permanent_uuid: "7ed1c497e3454a6eb54f4bcfda8552de" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 41485 } } peers { permanent_uuid: "d45fe0ed8f6b4ddb9b3f31f14ba18ace" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 41805 } }
I20260806 02:59:38.577504 7297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:38.576668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:36051 (local address 127.31.74.193:49063)
0806 02:59:38.576983 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:38.577318 (+ 335us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:36051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156}
I20260806 02:59:38.580937 6516 catalog_manager.cc:5495] T a7d896e3a3ba439598e3bf2e8e576bdf P 140e7485da14480cb28af227351c865f reported cstate change: term changed from 0 to 1, leader changed from <none> to 140e7485da14480cb28af227351c865f (127.31.74.193). New cstate: current_term: 1 leader_uuid: "140e7485da14480cb28af227351c865f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "140e7485da14480cb28af227351c865f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 46705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ed1c497e3454a6eb54f4bcfda8552de" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 41485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d45fe0ed8f6b4ddb9b3f31f14ba18ace" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 41805 } health_report { overall_health: UNKNOWN } } }
I20260806 02:59:38.615686 7284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:38.609164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55764 (local address 127.31.74.193:46705)
0806 02:59:38.609449 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:38.609454 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:38.609473 (+ 19us) server_negotiation.cc:407] Connection header received
0806 02:59:38.609721 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:38.609726 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:38.609952 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:38.610112 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:59:38.610767 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:38.611658 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:38.613086 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:38.613535 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:38.613686 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:38.614291 (+ 605us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:59:38.614986 (+ 695us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:59:38.615184 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:38.615291 (+ 107us) server_negotiation.cc:299] Negotiation successful
0806 02:59:38.615388 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260806 02:59:38.624305 7000 raft_consensus.cc:1261] T a7d896e3a3ba439598e3bf2e8e576bdf P d45fe0ed8f6b4ddb9b3f31f14ba18ace [term 1 FOLLOWER]: Refusing update from remote peer 140e7485da14480cb28af227351c865f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:59:38.624305 7144 raft_consensus.cc:1261] T a7d896e3a3ba439598e3bf2e8e576bdf P 7ed1c497e3454a6eb54f4bcfda8552de [term 1 FOLLOWER]: Refusing update from remote peer 140e7485da14480cb28af227351c865f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:59:38.626322 7295 consensus_queue.cc:1035] T a7d896e3a3ba439598e3bf2e8e576bdf P 140e7485da14480cb28af227351c865f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d45fe0ed8f6b4ddb9b3f31f14ba18ace" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:59:38.626948 7292 consensus_queue.cc:1035] T a7d896e3a3ba439598e3bf2e8e576bdf P 140e7485da14480cb28af227351c865f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ed1c497e3454a6eb54f4bcfda8552de" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 41485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:59:38.630786 7300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:38.630027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:36051 (local address 127.31.74.194:45291)
0806 02:59:38.630298 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:38.630476 (+ 178us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:36051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260806 02:59:38.631748 7302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:38.630747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:36051 (local address 127.31.74.195:46905)
0806 02:59:38.631112 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:38.631346 (+ 234us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:36051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155}
I20260806 02:59:38.637224 7307 mvcc.cc:205] Tried to move back new op lower bound from 7315395291629092864 to 7315395291454128128. Current Snapshot: MvccSnapshot[applied={T|T < 7315395291629092864}]
I20260806 02:59:38.639987 7302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:38.639232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:36051 (local address 127.31.74.195:45267)
0806 02:59:38.639576 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:38.639714 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:36051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181}
I20260806 02:59:38.642621 7297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:38.641931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:36051 (local address 127.31.74.193:39951)
0806 02:59:38.642267 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:38.642401 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:36051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260806 02:59:38.648447 7300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:38.647970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:36051 (local address 127.31.74.194:57361)
0806 02:59:38.648147 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:38.648265 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:36051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260806 02:59:38.660444 7297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:38.660038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:36051 (local address 127.31.74.193:36515)
0806 02:59:38.660207 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:38.660293 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:36051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260806 02:59:38.675473 6517 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:59:38.675822 6517 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:59:38.683681 6517 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 001d2d57ebfd4d6aa7d81fde49a50231: Sending DeleteTablet for 3 replicas of tablet a7d896e3a3ba439598e3bf2e8e576bdf
I20260806 02:59:38.684782 6825 tablet_service.cc:1514] Processing DeleteTablet for tablet a7d896e3a3ba439598e3bf2e8e576bdf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:59:38 UTC) from {username='oryx', principal='oryx/127.31.74.253@KRBTEST.COM'} at 127.0.0.1:55758
I20260806 02:59:38.684996 7124 tablet_service.cc:1514] Processing DeleteTablet for tablet a7d896e3a3ba439598e3bf2e8e576bdf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:59:38 UTC) from {username='oryx', principal='oryx/127.31.74.253@KRBTEST.COM'} at 127.0.0.1:57762
I20260806 02:59:38.686038 6979 tablet_service.cc:1514] Processing DeleteTablet for tablet a7d896e3a3ba439598e3bf2e8e576bdf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:59:38 UTC) from {username='oryx', principal='oryx/127.31.74.253@KRBTEST.COM'} at 127.0.0.1:38032
I20260806 02:59:38.686807 7317 tablet_replica.cc:330] T a7d896e3a3ba439598e3bf2e8e576bdf P 140e7485da14480cb28af227351c865f: stopping tablet replica
I20260806 02:59:38.687036 7317 raft_consensus.cc:2229] T a7d896e3a3ba439598e3bf2e8e576bdf P 140e7485da14480cb28af227351c865f [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:59:38.687412 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 6689
I20260806 02:59:38.687767 7317 raft_consensus.cc:2258] T a7d896e3a3ba439598e3bf2e8e576bdf P 140e7485da14480cb28af227351c865f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:59:38.688964 7317 ts_tablet_manager.cc:1905] T a7d896e3a3ba439598e3bf2e8e576bdf P 140e7485da14480cb28af227351c865f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:59:38.699419 7319 tablet_replica.cc:330] T a7d896e3a3ba439598e3bf2e8e576bdf P d45fe0ed8f6b4ddb9b3f31f14ba18ace: stopping tablet replica
I20260806 02:59:38.699862 7319 raft_consensus.cc:2229] T a7d896e3a3ba439598e3bf2e8e576bdf P d45fe0ed8f6b4ddb9b3f31f14ba18ace [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:59:38.700006 7317 ts_tablet_manager.cc:1918] T a7d896e3a3ba439598e3bf2e8e576bdf P 140e7485da14480cb28af227351c865f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:59:38.700228 7317 log.cc:1196] T a7d896e3a3ba439598e3bf2e8e576bdf P 140e7485da14480cb28af227351c865f: Deleting WAL directory at /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-0/wal/wals/a7d896e3a3ba439598e3bf2e8e576bdf
I20260806 02:59:38.700508 7319 raft_consensus.cc:2258] T a7d896e3a3ba439598e3bf2e8e576bdf P d45fe0ed8f6b4ddb9b3f31f14ba18ace [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:59:38.700829 7317 ts_tablet_manager.cc:1939] T a7d896e3a3ba439598e3bf2e8e576bdf P 140e7485da14480cb28af227351c865f: Deleting consensus metadata
I20260806 02:59:38.701715 7319 ts_tablet_manager.cc:1905] T a7d896e3a3ba439598e3bf2e8e576bdf P d45fe0ed8f6b4ddb9b3f31f14ba18ace: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:59:38.702657 6501 catalog_manager.cc:4841] TS 140e7485da14480cb28af227351c865f (127.31.74.193:46705): tablet a7d896e3a3ba439598e3bf2e8e576bdf (table test-table [id=674823d99f03425ba78981f8b347f4e6]) successfully deleted
I20260806 02:59:38.707978 7319 ts_tablet_manager.cc:1918] T a7d896e3a3ba439598e3bf2e8e576bdf P d45fe0ed8f6b4ddb9b3f31f14ba18ace: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:59:38.708339 7319 log.cc:1196] T a7d896e3a3ba439598e3bf2e8e576bdf P d45fe0ed8f6b4ddb9b3f31f14ba18ace: Deleting WAL directory at /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-1/wal/wals/a7d896e3a3ba439598e3bf2e8e576bdf
I20260806 02:59:38.708438 7318 tablet_replica.cc:330] T a7d896e3a3ba439598e3bf2e8e576bdf P 7ed1c497e3454a6eb54f4bcfda8552de: stopping tablet replica
I20260806 02:59:38.708717 7318 raft_consensus.cc:2229] T a7d896e3a3ba439598e3bf2e8e576bdf P 7ed1c497e3454a6eb54f4bcfda8552de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:59:38.709321 7319 ts_tablet_manager.cc:1939] T a7d896e3a3ba439598e3bf2e8e576bdf P d45fe0ed8f6b4ddb9b3f31f14ba18ace: Deleting consensus metadata
I20260806 02:59:38.709107 7318 raft_consensus.cc:2258] T a7d896e3a3ba439598e3bf2e8e576bdf P 7ed1c497e3454a6eb54f4bcfda8552de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:59:38.710687 7318 ts_tablet_manager.cc:1905] T a7d896e3a3ba439598e3bf2e8e576bdf P 7ed1c497e3454a6eb54f4bcfda8552de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:59:38.711611 6504 catalog_manager.cc:4841] TS d45fe0ed8f6b4ddb9b3f31f14ba18ace (127.31.74.194:41805): tablet a7d896e3a3ba439598e3bf2e8e576bdf (table test-table [id=674823d99f03425ba78981f8b347f4e6]) successfully deleted
I20260806 02:59:38.719467 7318 ts_tablet_manager.cc:1918] T a7d896e3a3ba439598e3bf2e8e576bdf P 7ed1c497e3454a6eb54f4bcfda8552de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:59:38.719655 7318 log.cc:1196] T a7d896e3a3ba439598e3bf2e8e576bdf P 7ed1c497e3454a6eb54f4bcfda8552de: Deleting WAL directory at /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785985105255475-32043-0/minicluster-data/ts-2/wal/wals/a7d896e3a3ba439598e3bf2e8e576bdf
I20260806 02:59:38.720130 7318 ts_tablet_manager.cc:1939] T a7d896e3a3ba439598e3bf2e8e576bdf P 7ed1c497e3454a6eb54f4bcfda8552de: Deleting consensus metadata
I20260806 02:59:38.721354 6501 catalog_manager.cc:4841] TS 7ed1c497e3454a6eb54f4bcfda8552de (127.31.74.195:41485): tablet a7d896e3a3ba439598e3bf2e8e576bdf (table test-table [id=674823d99f03425ba78981f8b347f4e6]) successfully deleted
I20260806 02:59:38.732831 7284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:38.688417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55780 (local address 127.31.74.193:46705)
0806 02:59:38.689201 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:38.689208 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:38.707347 (+ 18139us) server_negotiation.cc:407] Connection header received
0806 02:59:38.707624 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:38.707630 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:38.708109 (+ 479us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:38.708343 (+ 234us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:38.712303 (+ 3960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:38.713730 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:38.715931 (+ 2201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:38.716775 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:38.717339 (+ 564us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:38.721257 (+ 3918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:38.721301 (+ 44us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:38.721323 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:38.721379 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:38.728375 (+ 6996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:38.729552 (+ 1177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:38.729566 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:38.729569 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:38.729798 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:38.730320 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:38.730324 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:38.730326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:38.730867 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:38.731152 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:38.731610 (+ 458us) server_negotiation.cc:299] Negotiation successful
0806 02:59:38.731816 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628}
I20260806 02:59:38.733804 6886 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:59:38.860419 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 6689
I20260806 02:59:38.887748 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 6903
I20260806 02:59:38.905803 7287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:38.888832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38046 (local address 127.31.74.194:41805)
0806 02:59:38.889079 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:38.889083 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:38.889103 (+ 20us) server_negotiation.cc:407] Connection header received
0806 02:59:38.889364 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:38.889369 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:38.889449 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:38.889753 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:38.891382 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:38.892486 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:38.894226 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:38.894809 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:38.894929 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:38.897721 (+ 2792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:38.897740 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:38.897757 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:38.897811 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:38.902133 (+ 4322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:38.903387 (+ 1254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:38.903396 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:38.903399 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:38.903600 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:38.904096 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:38.904101 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:38.904103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:38.904562 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:38.904877 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:38.905139 (+ 262us) server_negotiation.cc:299] Negotiation successful
0806 02:59:38.905342 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260806 02:59:38.906680 7040 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:59:39.032678 7323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:39.031819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:36051 (local address 127.0.0.1:38212)
0806 02:59:39.032395 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:39.032491 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:36051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":339,"threads_started":1}
I20260806 02:59:39.038035 7323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:39.037581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:36051 (local address 127.0.0.1:38226)
0806 02:59:39.037777 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:39.037877 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:36051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260806 02:59:39.044011 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 6903
I20260806 02:59:39.073055 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 7058
I20260806 02:59:39.088657 7325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:39.073920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57770 (local address 127.31.74.195:41485)
0806 02:59:39.074407 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:39.074412 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:39.074428 (+ 16us) server_negotiation.cc:407] Connection header received
0806 02:59:39.074486 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:39.074490 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:39.074555 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:39.074713 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:39.076187 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:39.077126 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:39.078484 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:39.079080 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:39.079239 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:39.081895 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:39.081910 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:39.081926 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:39.081959 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:39.085225 (+ 3266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:39.086039 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:39.086047 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:39.086050 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:39.086200 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:39.086942 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:39.086947 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:39.086949 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:39.087397 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:39.087656 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:39.088069 (+ 413us) server_negotiation.cc:299] Negotiation successful
0806 02:59:39.088310 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":256,"threads_started":1}
I20260806 02:59:39.089448 7184 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:59:39.193218 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 7058
I20260806 02:59:39.213680 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 6377
I20260806 02:59:39.214956 6548 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:59:39.350842 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 6377
I20260806 02:59:39.377835 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 6483
I20260806 02:59:39.379091 6741 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:59:39.502049 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 6483
2026-08-06T02:59:39Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (11287 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:59:39 dist-test-slave-dhph krb5kdc[7331](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:59:39 dist-test-slave-dhph krb5kdc[7331](info): set up 2 sockets
Aug 06 02:59:39 dist-test-slave-dhph krb5kdc[7331](info): commencing operation
krb5kdc: starting...
W20260806 02:59:40.774870 32043 mini_kdc.cc:121] Time spent starting KDC: real 1.216s user 0.003s sys 0.043s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/krb5kdc/test-user.keytab.
Aug 06 02:59:40 dist-test-slave-dhph krb5kdc[7331](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985180, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:59:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:59:41Z Disabled control of system clock
WARNING: no policy specified for oryx/127.31.74.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.74.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.74.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/krb5kdc/oryx_127.31.74.254.keytab.
Entry for principal oryx/127.31.74.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/krb5kdc/oryx_127.31.74.254.keytab.
I20260806 02:59:41.104486 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:40379
--webserver_interface=127.31.74.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/krb5kdc/oryx_127.31.74.254.keytab
--principal=oryx/127.31.74.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:37531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.74.254:40379
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:59:41.489951 7344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:59:41.490295 7344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:59:41.490486 7344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:59:41.498996 7344 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:59:41.499130 7344 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260806 02:59:41.499174 7344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:59:41.499308 7344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:59:41.499383 7344 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:59:41.499495 7344 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:59:41.510111 7344 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:37531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.74.254:40379
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/krb5kdc/oryx_127.31.74.254.keytab
--principal=oryx/127.31.74.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:40379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.74.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:59:41.511611 7344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:59:41.513013 7344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:59:41.528534 7349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:41.528683 7350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:41.528980 7352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:59:41.529882 7344 server_base.cc:1034] running on GCE node
I20260806 02:59:41.530884 7344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:59:41.532558 7344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:59:41.534022 7344 hybrid_clock.cc:648] HybridClock initialized: now 1785985181533880 us; error 111 us; skew 500 ppm
I20260806 02:59:41.534512 7344 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:59:41 dist-test-slave-dhph krb5kdc[7331](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985181, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.74.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:59:41.538506 7344 init.cc:377] Logged in from keytab as oryx/127.31.74.254@KRBTEST.COM (short username oryx)
I20260806 02:59:41.540529 7344 webserver.cc:460] Webserver started at http://127.31.74.254:33669/ using document root <none> and password file <none>
I20260806 02:59:41.541417 7344 fs_manager.cc:359] Metadata directory not provided
I20260806 02:59:41.541574 7344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:59:41.541955 7344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:59:41.545482 7344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/minicluster-data/master-0/data/instance:
uuid: "8643cb88550544428f8c7eee4349597e"
format_stamp: "Formatted at 2026-08-06 02:59:41 on dist-test-slave-dhph"
server_key: "d0c0bcdca52c60dae593393b03f77169"
server_key_iv: "0d193a2b017539d549d0b8cbc8714f5b"
server_key_version: "encryptionkey@0"
I20260806 02:59:41.546649 7344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/minicluster-data/master-0/wal/instance:
uuid: "8643cb88550544428f8c7eee4349597e"
format_stamp: "Formatted at 2026-08-06 02:59:41 on dist-test-slave-dhph"
server_key: "d0c0bcdca52c60dae593393b03f77169"
server_key_iv: "0d193a2b017539d549d0b8cbc8714f5b"
server_key_version: "encryptionkey@0"
I20260806 02:59:41.552640 7344 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.003s
I20260806 02:59:41.557305 7359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:41.559070 7344 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 02:59:41.559396 7344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/minicluster-data/master-0/wal
uuid: "8643cb88550544428f8c7eee4349597e"
format_stamp: "Formatted at 2026-08-06 02:59:41 on dist-test-slave-dhph"
server_key: "d0c0bcdca52c60dae593393b03f77169"
server_key_iv: "0d193a2b017539d549d0b8cbc8714f5b"
server_key_version: "encryptionkey@0"
I20260806 02:59:41.559659 7344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:59:41.583225 7344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:59:41.588472 7344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:59:41.588809 7344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:59:41.616133 7344 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.254:40379
I20260806 02:59:41.616171 7420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.254:40379 every 8 connection(s)
I20260806 02:59:41.618690 7344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
I20260806 02:59:41.623528 7421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:59:41.628957 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 7344
I20260806 02:59:41.629202 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/minicluster-data/master-0/wal/instance
I20260806 02:59:41.630122 32043 external_mini_cluster.cc:1428] Setting key faea96f68f064af0cfb9131129dd5b43
I20260806 02:59:41.641597 7421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8643cb88550544428f8c7eee4349597e: Bootstrap starting.
I20260806 02:59:41.646307 7421 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8643cb88550544428f8c7eee4349597e: Neither blocks nor log segments found. Creating new log.
I20260806 02:59:41.648074 7421 log.cc:824] T 00000000000000000000000000000000 P 8643cb88550544428f8c7eee4349597e: Log is configured to *not* fsync() on all Append() calls
Aug 06 02:59:41 dist-test-slave-dhph krb5kdc[7331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985180, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM
I20260806 02:59:41.652670 7421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8643cb88550544428f8c7eee4349597e: No bootstrap required, opened a new log
I20260806 02:59:41.664361 7424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:41.632693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44446 (local address 127.31.74.254:40379)
0806 02:59:41.633641 (+ 948us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:41.633655 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:41.633699 (+ 44us) server_negotiation.cc:407] Connection header received
0806 02:59:41.635723 (+ 2024us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:41.635752 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:41.636251 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:41.636646 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:41.639302 (+ 2656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:41.640809 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:41.642976 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:41.644129 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:41.644291 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:41.651796 (+ 7505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:41.651813 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:41.651837 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:41.651905 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:41.657124 (+ 5219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:41.658359 (+ 1235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:41.658374 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:41.658412 (+ 38us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:41.658697 (+ 285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:41.659940 (+ 1243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:41.659945 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:41.659948 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:41.660955 (+ 1007us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:41.661342 (+ 387us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:41.661947 (+ 605us) server_negotiation.cc:299] Negotiation successful
0806 02:59:41.662568 (+ 621us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":675,"thread_start_us":432,"threads_started":1}
I20260806 02:59:41.664870 7421 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8643cb88550544428f8c7eee4349597e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8643cb88550544428f8c7eee4349597e" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 40379 } }
I20260806 02:59:41.665367 7421 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8643cb88550544428f8c7eee4349597e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:59:41.665486 7421 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8643cb88550544428f8c7eee4349597e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8643cb88550544428f8c7eee4349597e, State: Initialized, Role: FOLLOWER
I20260806 02:59:41.666184 7421 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8643cb88550544428f8c7eee4349597e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8643cb88550544428f8c7eee4349597e" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 40379 } }
I20260806 02:59:41.666445 7421 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8643cb88550544428f8c7eee4349597e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:59:41.666636 7421 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8643cb88550544428f8c7eee4349597e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:59:41.666890 7421 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8643cb88550544428f8c7eee4349597e [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:59:41.669698 7421 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8643cb88550544428f8c7eee4349597e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8643cb88550544428f8c7eee4349597e" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 40379 } }
I20260806 02:59:41.671110 7421 leader_election.cc:302] T 00000000000000000000000000000000 P 8643cb88550544428f8c7eee4349597e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8643cb88550544428f8c7eee4349597e; no voters:
I20260806 02:59:41.671875 7421 leader_election.cc:288] T 00000000000000000000000000000000 P 8643cb88550544428f8c7eee4349597e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:59:41.672035 7426 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8643cb88550544428f8c7eee4349597e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:59:41.672829 7426 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8643cb88550544428f8c7eee4349597e [term 1 LEADER]: Becoming Leader. State: Replica: 8643cb88550544428f8c7eee4349597e, State: Running, Role: LEADER
I20260806 02:59:41.673496 7426 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8643cb88550544428f8c7eee4349597e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8643cb88550544428f8c7eee4349597e" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 40379 } }
I20260806 02:59:41.674115 7421 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8643cb88550544428f8c7eee4349597e [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:59:41.677170 7428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8643cb88550544428f8c7eee4349597e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8643cb88550544428f8c7eee4349597e. Latest consensus state: current_term: 1 leader_uuid: "8643cb88550544428f8c7eee4349597e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8643cb88550544428f8c7eee4349597e" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 40379 } } }
I20260806 02:59:41.677596 7428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8643cb88550544428f8c7eee4349597e [sys.catalog]: This master's current role is: LEADER
I20260806 02:59:41.678853 7427 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8643cb88550544428f8c7eee4349597e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8643cb88550544428f8c7eee4349597e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8643cb88550544428f8c7eee4349597e" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 40379 } } }
I20260806 02:59:41.679344 7427 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8643cb88550544428f8c7eee4349597e [sys.catalog]: This master's current role is: LEADER
I20260806 02:59:41.680673 7436 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:59:41.686543 7436 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:59:41.697369 7436 catalog_manager.cc:1347] Generated new cluster ID: 8b2f65a88cc94b5497e1a0693454c24a
I20260806 02:59:41.697544 7436 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:59:41.705456 7436 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:59:41.707443 7436 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:59:41.717175 7436 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8643cb88550544428f8c7eee4349597e: Generated new TSK 0
I20260806 02:59:41.718394 7436 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.31.74.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.74.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.74.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/krb5kdc/oryx_127.31.74.193.keytab.
Entry for principal oryx/127.31.74.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/krb5kdc/oryx_127.31.74.193.keytab.
I20260806 02:59:41.803319 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.193:0
--local_ip_for_outbound_sockets=127.31.74.193
--webserver_interface=127.31.74.193
--webserver_port=0
--tserver_master_addrs=127.31.74.254:40379
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/krb5kdc/oryx_127.31.74.193.keytab
--principal=oryx/127.31.74.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:37531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:59:42.178529 7449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:59:42.178999 7449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:59:42.179173 7449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:59:42.186949 7449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:59:42.187131 7449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:59:42.187286 7449 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 02:59:42.187438 7449 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.193
I20260806 02:59:42.199966 7449 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:37531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/krb5kdc/oryx_127.31.74.193.keytab
--principal=oryx/127.31.74.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.74.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.74.254:40379
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.193
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:59:42.201432 7449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:59:42.202972 7449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:59:42.219218 7455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:42.219205 7454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:42.219173 7457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:59:42.220510 7449 server_base.cc:1034] running on GCE node
I20260806 02:59:42.221252 7449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:59:42.222270 7449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:59:42.223558 7449 hybrid_clock.cc:648] HybridClock initialized: now 1785985182223444 us; error 73 us; skew 500 ppm
I20260806 02:59:42.224080 7449 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:59:42 dist-test-slave-dhph krb5kdc[7331](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985182, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.74.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:59:42.228289 7449 init.cc:377] Logged in from keytab as oryx/127.31.74.193@KRBTEST.COM (short username oryx)
I20260806 02:59:42.230224 7449 webserver.cc:460] Webserver started at http://127.31.74.193:33817/ using document root <none> and password file <none>
I20260806 02:59:42.231099 7449 fs_manager.cc:359] Metadata directory not provided
I20260806 02:59:42.231257 7449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:59:42.231640 7449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:59:42.234993 7449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/minicluster-data/ts-0/data/instance:
uuid: "b98ed2c913804afe994cafecc1994347"
format_stamp: "Formatted at 2026-08-06 02:59:42 on dist-test-slave-dhph"
server_key: "cf4deca3d20752241a1fc32a5819b0f2"
server_key_iv: "727b96602489655fbc578b1fb23466c2"
server_key_version: "encryptionkey@0"
I20260806 02:59:42.235954 7449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/minicluster-data/ts-0/wal/instance:
uuid: "b98ed2c913804afe994cafecc1994347"
format_stamp: "Formatted at 2026-08-06 02:59:42 on dist-test-slave-dhph"
server_key: "cf4deca3d20752241a1fc32a5819b0f2"
server_key_iv: "727b96602489655fbc578b1fb23466c2"
server_key_version: "encryptionkey@0"
I20260806 02:59:42.241818 7449 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 02:59:42.246382 7464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:42.248199 7449 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 02:59:42.248477 7449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/minicluster-data/ts-0/wal
uuid: "b98ed2c913804afe994cafecc1994347"
format_stamp: "Formatted at 2026-08-06 02:59:42 on dist-test-slave-dhph"
server_key: "cf4deca3d20752241a1fc32a5819b0f2"
server_key_iv: "727b96602489655fbc578b1fb23466c2"
server_key_version: "encryptionkey@0"
I20260806 02:59:42.248768 7449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:59:42.289076 7449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:59:42.293980 7449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:59:42.294304 7449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:59:42.297681 7449 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:59:42.297817 7449 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:42.298111 7449 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:59:42.298208 7449 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 02:59:42 dist-test-slave-dhph krb5kdc[7331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985182, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.74.193@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM
I20260806 02:59:42.326924 7473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:42.298622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.193:55293 (local address 127.31.74.254:40379)
0806 02:59:42.299275 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:42.299281 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:42.300394 (+ 1113us) server_negotiation.cc:407] Connection header received
0806 02:59:42.302670 (+ 2276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:42.302677 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:42.302795 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:42.302928 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:42.305715 (+ 2787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:42.307109 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:42.310385 (+ 3276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:42.311298 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:42.311547 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:42.317294 (+ 5747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:42.317305 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:42.317313 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:42.317470 (+ 157us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:42.320928 (+ 3458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:42.322338 (+ 1410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:42.322347 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:42.322354 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:42.322536 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:42.323866 (+ 1330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:42.323876 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:42.323882 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:42.324357 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:42.324540 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:42.325944 (+ 1404us) server_negotiation.cc:299] Negotiation successful
0806 02:59:42.326372 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":409,"threads_started":1}
I20260806 02:59:42.328715 7474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:42.299164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:40379 (local address 127.31.74.193:55293)
0806 02:59:42.300133 (+ 969us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:42.300184 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:42.302338 (+ 2154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:42.303258 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:42.303278 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:42.304256 (+ 978us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:59:42.305399 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:42.305437 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:42.307496 (+ 2059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:42.307501 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:42.310190 (+ 2689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:42.310207 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:42.311716 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:42.311731 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:42.311987 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:42.312937 (+ 950us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:59:42.312962 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:59:42.316924 (+ 3962us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:59:42.321151 (+ 4227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:42.321167 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:42.321205 (+ 38us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:42.322134 (+ 929us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:42.322721 (+ 587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:42.322730 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:42.322737 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:42.323676 (+ 939us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:42.324656 (+ 980us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:59:42.324669 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:59:42.325338 (+ 669us) client_negotiation.cc:769] Sending connection context
0806 02:59:42.325761 (+ 423us) client_negotiation.cc:240] Negotiation successful
0806 02:59:42.326601 (+ 840us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":704,"thread_start_us":441,"threads_started":1}
I20260806 02:59:42.355472 7473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:42.336983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.193:43229 (local address 127.31.74.254:40379)
0806 02:59:42.337216 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:42.337224 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:42.338150 (+ 926us) server_negotiation.cc:407] Connection header received
0806 02:59:42.338391 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:42.338396 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:42.338480 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:42.338614 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:42.340473 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:42.341841 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:42.344027 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:42.345004 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:42.345187 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:42.348237 (+ 3050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:42.348247 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:42.348257 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:42.348295 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:42.352095 (+ 3800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:42.352933 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:42.352948 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:42.352951 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:42.353095 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:42.353531 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:42.353540 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:42.353544 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:42.353889 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:42.354065 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:42.354735 (+ 670us) server_negotiation.cc:299] Negotiation successful
0806 02:59:42.355064 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260806 02:59:42.355531 7557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:42.337108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:40379 (local address 127.31.74.193:43229)
0806 02:59:42.338028 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:42.338047 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:42.338232 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:42.338662 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:42.338666 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:42.339581 (+ 915us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:59:42.340235 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:42.340256 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:42.342069 (+ 1813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:42.342073 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:42.343849 (+ 1776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:42.343865 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:42.345260 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:42.345267 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:42.345419 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:42.346317 (+ 898us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:59:42.346360 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:59:42.347980 (+ 1620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:59:42.352274 (+ 4294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:42.352281 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:42.352286 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:42.352779 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:42.353235 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:42.353241 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:42.353245 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:42.353395 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:42.354021 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:59:42.354027 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:59:42.354259 (+ 232us) client_negotiation.cc:769] Sending connection context
0806 02:59:42.354423 (+ 164us) client_negotiation.cc:240] Negotiation successful
0806 02:59:42.354973 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":744,"thread_start_us":528,"threads_started":1}
I20260806 02:59:42.361238 7449 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.193:35105
I20260806 02:59:42.361327 7587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.193:35105 every 8 connection(s)
I20260806 02:59:42.362495 7449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
I20260806 02:59:42.365156 7588 heartbeater.cc:347] Connected to a master server at 127.31.74.254:40379
I20260806 02:59:42.365391 7588 heartbeater.cc:464] Registering TS with master...
I20260806 02:59:42.366086 7588 heartbeater.cc:511] Master 127.31.74.254:40379 requested a full tablet report, sending...
I20260806 02:59:42.368511 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 7449
I20260806 02:59:42.368757 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/minicluster-data/ts-0/wal/instance
I20260806 02:59:42.368817 7376 ts_manager.cc:194] Registered new tserver with Master: b98ed2c913804afe994cafecc1994347 (127.31.74.193:35105)
I20260806 02:59:42.369961 32043 external_mini_cluster.cc:1428] Setting key e567c689f82d780e3035e90072339ad8
I20260806 02:59:42.371384 7376 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.74.193@KRBTEST.COM'} at 127.31.74.193:55293
WARNING: no policy specified for oryx/127.31.74.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.74.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.74.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/krb5kdc/oryx_127.31.74.194.keytab.
Entry for principal oryx/127.31.74.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/krb5kdc/oryx_127.31.74.194.keytab.
I20260806 02:59:42.452764 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.194:0
--local_ip_for_outbound_sockets=127.31.74.194
--webserver_interface=127.31.74.194
--webserver_port=0
--tserver_master_addrs=127.31.74.254:40379
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/krb5kdc/oryx_127.31.74.194.keytab
--principal=oryx/127.31.74.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:37531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:59:42.846530 7592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:59:42.846935 7592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:59:42.847123 7592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:59:42.856030 7592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:59:42.856316 7592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:59:42.856680 7592 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 02:59:42.857050 7592 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.194
I20260806 02:59:42.868969 7592 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:37531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/krb5kdc/oryx_127.31.74.194.keytab
--principal=oryx/127.31.74.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.74.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.74.254:40379
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.194
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:59:42.870707 7592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:59:42.872391 7592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:59:42.888311 7598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:42.888530 7597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:42.888463 7600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:59:42.889407 7592 server_base.cc:1034] running on GCE node
I20260806 02:59:42.890169 7592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:59:42.891343 7592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:59:42.892659 7592 hybrid_clock.cc:648] HybridClock initialized: now 1785985182892540 us; error 75 us; skew 500 ppm
I20260806 02:59:42.893083 7592 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:59:42 dist-test-slave-dhph krb5kdc[7331](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985182, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.74.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:59:42.897063 7592 init.cc:377] Logged in from keytab as oryx/127.31.74.194@KRBTEST.COM (short username oryx)
I20260806 02:59:42.898926 7592 webserver.cc:460] Webserver started at http://127.31.74.194:37099/ using document root <none> and password file <none>
I20260806 02:59:42.899744 7592 fs_manager.cc:359] Metadata directory not provided
I20260806 02:59:42.899912 7592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:59:42.900305 7592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:59:42.903828 7592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/minicluster-data/ts-1/data/instance:
uuid: "2fd6a06c3426465484a699dedffbf3b4"
format_stamp: "Formatted at 2026-08-06 02:59:42 on dist-test-slave-dhph"
server_key: "4aa1fac6d718d43a071a0bcb52b440c3"
server_key_iv: "8857993eab13d24752ac522921ad8751"
server_key_version: "encryptionkey@0"
I20260806 02:59:42.904788 7592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/minicluster-data/ts-1/wal/instance:
uuid: "2fd6a06c3426465484a699dedffbf3b4"
format_stamp: "Formatted at 2026-08-06 02:59:42 on dist-test-slave-dhph"
server_key: "4aa1fac6d718d43a071a0bcb52b440c3"
server_key_iv: "8857993eab13d24752ac522921ad8751"
server_key_version: "encryptionkey@0"
I20260806 02:59:42.912578 7592 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260806 02:59:42.917062 7607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:42.918995 7592 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260806 02:59:42.919226 7592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/minicluster-data/ts-1/wal
uuid: "2fd6a06c3426465484a699dedffbf3b4"
format_stamp: "Formatted at 2026-08-06 02:59:42 on dist-test-slave-dhph"
server_key: "4aa1fac6d718d43a071a0bcb52b440c3"
server_key_iv: "8857993eab13d24752ac522921ad8751"
server_key_version: "encryptionkey@0"
I20260806 02:59:42.919523 7592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:59:42.936343 7592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:59:42.941901 7592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:59:42.942250 7592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:59:42.945470 7592 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:59:42.945596 7592 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:42.945793 7592 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:59:42.945873 7592 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 02:59:42 dist-test-slave-dhph krb5kdc[7331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985182, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.74.194@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM
I20260806 02:59:42.973704 7615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:42.945884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.194:36411 (local address 127.31.74.254:40379)
0806 02:59:42.946440 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:42.946449 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:42.948104 (+ 1655us) server_negotiation.cc:407] Connection header received
0806 02:59:42.950730 (+ 2626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:42.950737 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:42.950847 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:42.950984 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:42.954302 (+ 3318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:42.956088 (+ 1786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:42.958851 (+ 2763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:42.959442 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:42.959608 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:42.965282 (+ 5674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:42.965288 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:42.965291 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:42.965332 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:42.969132 (+ 3800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:42.970017 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:42.970022 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:42.970025 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:42.970169 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:42.970718 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:42.970773 (+ 55us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:42.970777 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:42.971253 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:42.971489 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:42.972647 (+ 1158us) server_negotiation.cc:299] Negotiation successful
0806 02:59:42.973037 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":311,"threads_started":1}
I20260806 02:59:42.974347 7617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:42.946648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:40379 (local address 127.31.74.194:36411)
0806 02:59:42.947895 (+ 1247us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:42.947957 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:42.950293 (+ 2336us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:42.951461 (+ 1168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:42.951479 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:42.952668 (+ 1189us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:59:42.954004 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:42.954039 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:42.956453 (+ 2414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:42.956465 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:42.958560 (+ 2095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:42.958637 (+ 77us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:42.959752 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:42.959761 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:42.960020 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:42.960944 (+ 924us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:59:42.960972 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:59:42.964963 (+ 3991us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:59:42.969330 (+ 4367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:42.969341 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:42.969367 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:42.969858 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:42.970304 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:42.970309 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:42.970313 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:42.970552 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:42.971456 (+ 904us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:59:42.971465 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:59:42.972077 (+ 612us) client_negotiation.cc:769] Sending connection context
0806 02:59:42.972475 (+ 398us) client_negotiation.cc:240] Negotiation successful
0806 02:59:42.972997 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":982,"thread_start_us":518,"threads_started":1}
I20260806 02:59:43.007237 7615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:42.986364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.194:39729 (local address 127.31.74.254:40379)
0806 02:59:42.986705 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:42.986715 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:42.988184 (+ 1469us) server_negotiation.cc:407] Connection header received
0806 02:59:42.988413 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:42.988422 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:42.988527 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:42.988636 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:42.991685 (+ 3049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:42.992954 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:42.996301 (+ 3347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:42.997187 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:42.997338 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:43.000419 (+ 3081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:43.000430 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:43.000436 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:43.000490 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:43.003543 (+ 3053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:43.004448 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:43.004455 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:43.004461 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:43.004631 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:43.005235 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:43.005241 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:43.005247 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:43.005650 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:43.005775 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:43.006472 (+ 697us) server_negotiation.cc:299] Negotiation successful
0806 02:59:43.006801 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
I20260806 02:59:43.007637 7675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:42.987105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:40379 (local address 127.31.74.194:39729)
0806 02:59:42.988056 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:42.988075 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:42.988260 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:42.988870 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:42.988878 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:42.990367 (+ 1489us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:59:42.991291 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:42.991316 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:42.993209 (+ 1893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:42.993218 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:42.996004 (+ 2786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:42.996031 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:42.997775 (+ 1744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:42.997785 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:42.998059 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:42.999063 (+ 1004us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:59:42.999084 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:59:43.000137 (+ 1053us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:59:43.003692 (+ 3555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:43.003699 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:43.003705 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:43.004308 (+ 603us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:43.004790 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:43.004802 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:43.004809 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:43.005046 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:43.005793 (+ 747us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:59:43.005802 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:59:43.006089 (+ 287us) client_negotiation.cc:769] Sending connection context
0806 02:59:43.006312 (+ 223us) client_negotiation.cc:240] Negotiation successful
0806 02:59:43.006881 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":811,"thread_start_us":651,"threads_started":1}
I20260806 02:59:43.014685 7592 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.194:46395
I20260806 02:59:43.014765 7729 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.194:46395 every 8 connection(s)
I20260806 02:59:43.016685 7592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
I20260806 02:59:43.019681 7730 heartbeater.cc:347] Connected to a master server at 127.31.74.254:40379
I20260806 02:59:43.019997 7730 heartbeater.cc:464] Registering TS with master...
I20260806 02:59:43.020836 7730 heartbeater.cc:511] Master 127.31.74.254:40379 requested a full tablet report, sending...
I20260806 02:59:43.022203 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 7592
I20260806 02:59:43.022445 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/minicluster-data/ts-1/wal/instance
I20260806 02:59:43.022794 7375 ts_manager.cc:194] Registered new tserver with Master: 2fd6a06c3426465484a699dedffbf3b4 (127.31.74.194:46395)
I20260806 02:59:43.023021 32043 external_mini_cluster.cc:1428] Setting key 608bd0ecfd32fe102d3021e1789e6ae9
I20260806 02:59:43.023970 7375 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.74.194@KRBTEST.COM'} at 127.31.74.194:36411
WARNING: no policy specified for oryx/127.31.74.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.74.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.74.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/krb5kdc/oryx_127.31.74.195.keytab.
Entry for principal oryx/127.31.74.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/krb5kdc/oryx_127.31.74.195.keytab.
I20260806 02:59:43.091413 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.195:0
--local_ip_for_outbound_sockets=127.31.74.195
--webserver_interface=127.31.74.195
--webserver_port=0
--tserver_master_addrs=127.31.74.254:40379
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/krb5kdc/oryx_127.31.74.195.keytab
--principal=oryx/127.31.74.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:37531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:59:43.376001 7588 heartbeater.cc:503] Master 127.31.74.254:40379 was elected leader, sending a full tablet report...
W20260806 02:59:43.485054 7734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:59:43.485419 7734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:59:43.485592 7734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:59:43.494304 7734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:59:43.494501 7734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:59:43.494705 7734 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 02:59:43.494855 7734 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.195
I20260806 02:59:43.506217 7734 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:37531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/krb5kdc/oryx_127.31.74.195.keytab
--principal=oryx/127.31.74.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.74.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.74.254:40379
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.195
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:59:43.508280 7734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:59:43.509999 7734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:59:43.526360 7739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:43.526614 7742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:43.526872 7740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:59:43.527426 7734 server_base.cc:1034] running on GCE node
I20260806 02:59:43.528179 7734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:59:43.529247 7734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:59:43.530714 7734 hybrid_clock.cc:648] HybridClock initialized: now 1785985183530522 us; error 47 us; skew 500 ppm
I20260806 02:59:43.531327 7734 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:59:43 dist-test-slave-dhph krb5kdc[7331](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985183, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.74.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:59:43.541460 7734 init.cc:377] Logged in from keytab as oryx/127.31.74.195@KRBTEST.COM (short username oryx)
I20260806 02:59:43.543637 7734 webserver.cc:460] Webserver started at http://127.31.74.195:40849/ using document root <none> and password file <none>
I20260806 02:59:43.544423 7734 fs_manager.cc:359] Metadata directory not provided
I20260806 02:59:43.544567 7734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:59:43.544901 7734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:59:43.548873 7734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/minicluster-data/ts-2/data/instance:
uuid: "67c6964d17f443dcacfb6a01a0945be4"
format_stamp: "Formatted at 2026-08-06 02:59:43 on dist-test-slave-dhph"
server_key: "f94f1ad1e49c08c6262968b4f7921fb8"
server_key_iv: "cbf2ad7593eebca5e04ee2bf1560104f"
server_key_version: "encryptionkey@0"
I20260806 02:59:43.549760 7734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/minicluster-data/ts-2/wal/instance:
uuid: "67c6964d17f443dcacfb6a01a0945be4"
format_stamp: "Formatted at 2026-08-06 02:59:43 on dist-test-slave-dhph"
server_key: "f94f1ad1e49c08c6262968b4f7921fb8"
server_key_iv: "cbf2ad7593eebca5e04ee2bf1560104f"
server_key_version: "encryptionkey@0"
I20260806 02:59:43.555768 7734 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260806 02:59:43.560213 7749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:43.561897 7734 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:59:43.562153 7734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/minicluster-data/ts-2/wal
uuid: "67c6964d17f443dcacfb6a01a0945be4"
format_stamp: "Formatted at 2026-08-06 02:59:43 on dist-test-slave-dhph"
server_key: "f94f1ad1e49c08c6262968b4f7921fb8"
server_key_iv: "cbf2ad7593eebca5e04ee2bf1560104f"
server_key_version: "encryptionkey@0"
I20260806 02:59:43.562435 7734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:59:43.586917 7734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:59:43.591909 7734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:59:43.592185 7734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:59:43.595361 7734 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:59:43.595552 7734 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260806 02:59:43.596115 7734 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:59:43.596521 7734 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
Aug 06 02:59:43 dist-test-slave-dhph krb5kdc[7331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985183, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.74.195@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM
I20260806 02:59:43.626868 7758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:43.596986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.195:39609 (local address 127.31.74.254:40379)
0806 02:59:43.597854 (+ 868us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:43.597859 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:43.600614 (+ 2755us) server_negotiation.cc:407] Connection header received
0806 02:59:43.603454 (+ 2840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:43.603460 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:43.603570 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:43.603703 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:43.606745 (+ 3042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:43.607950 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:43.610357 (+ 2407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:43.611238 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:43.611567 (+ 329us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:43.617330 (+ 5763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:43.617337 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:43.617340 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:43.617383 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:43.621240 (+ 3857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:43.622694 (+ 1454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:43.622706 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:43.622711 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:43.623034 (+ 323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:43.624162 (+ 1128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:43.624171 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:43.624176 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:43.624702 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:43.624932 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:43.625990 (+ 1058us) server_negotiation.cc:299] Negotiation successful
0806 02:59:43.626297 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":748,"thread_start_us":594,"threads_started":1}
I20260806 02:59:43.628178 7760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:43.598007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:40379 (local address 127.31.74.195:39609)
0806 02:59:43.600363 (+ 2356us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:43.600434 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:43.603049 (+ 2615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:43.604208 (+ 1159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:43.604224 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:43.605251 (+ 1027us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:59:43.606461 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:43.606485 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:43.608221 (+ 1736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:43.608226 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:43.610090 (+ 1864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:43.610144 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:43.611717 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:43.611728 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:43.612157 (+ 429us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:43.613209 (+ 1052us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:59:43.613265 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:59:43.617015 (+ 3750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:59:43.621586 (+ 4571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:43.621601 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:43.621630 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:43.622333 (+ 703us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:43.623276 (+ 943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:43.623287 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:43.623291 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:43.623714 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:43.624917 (+ 1203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:59:43.624927 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:59:43.625526 (+ 599us) client_negotiation.cc:769] Sending connection context
0806 02:59:43.625858 (+ 332us) client_negotiation.cc:240] Negotiation successful
0806 02:59:43.626550 (+ 692us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1812,"thread_start_us":868,"threads_started":1}
I20260806 02:59:43.657578 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:43.639808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:40379 (local address 127.31.74.195:42523)
0806 02:59:43.640590 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:43.640613 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:43.640825 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:43.641560 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:43.641569 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:43.642424 (+ 855us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:59:43.643027 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:43.643043 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:43.644378 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:43.644383 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:43.646994 (+ 2611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:43.647022 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:43.647980 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:43.647987 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:43.648166 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:43.648865 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:59:43.648883 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:59:43.649671 (+ 788us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:59:43.653518 (+ 3847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:43.653522 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:43.653524 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:43.654003 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:43.654830 (+ 827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:43.654834 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:43.654836 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:43.654961 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:43.655625 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:59:43.655630 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:59:43.655926 (+ 296us) client_negotiation.cc:769] Sending connection context
0806 02:59:43.656114 (+ 188us) client_negotiation.cc:240] Negotiation successful
0806 02:59:43.656435 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":399,"threads_started":1}
I20260806 02:59:43.657671 7758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:43.639949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.195:42523 (local address 127.31.74.254:40379)
0806 02:59:43.640344 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:43.640357 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:43.640753 (+ 396us) server_negotiation.cc:407] Connection header received
0806 02:59:43.641002 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:43.641012 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:43.641142 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:43.641314 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:43.643220 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:43.644204 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:43.647271 (+ 3067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:43.647717 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:43.647884 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:43.649900 (+ 2016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:43.649905 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:43.649908 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:43.649943 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:43.653253 (+ 3310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:43.654225 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:43.654233 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:43.654238 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:43.654452 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:43.655091 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:43.655098 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:43.655101 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:43.655498 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:43.655660 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:43.656726 (+ 1066us) server_negotiation.cc:299] Negotiation successful
0806 02:59:43.657072 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260806 02:59:43.672236 7734 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.195:38549
I20260806 02:59:43.672284 7872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.195:38549 every 8 connection(s)
I20260806 02:59:43.674392 7734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
I20260806 02:59:43.676512 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 7734
I20260806 02:59:43.676785 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/minicluster-data/ts-2/wal/instance
I20260806 02:59:43.677426 32043 external_mini_cluster.cc:1428] Setting key d36530fbceb622ec0c03429eddb83592
I20260806 02:59:43.679090 7873 heartbeater.cc:347] Connected to a master server at 127.31.74.254:40379
I20260806 02:59:43.679497 7873 heartbeater.cc:464] Registering TS with master...
I20260806 02:59:43.680769 7873 heartbeater.cc:511] Master 127.31.74.254:40379 requested a full tablet report, sending...
I20260806 02:59:43.683694 7376 ts_manager.cc:194] Registered new tserver with Master: 67c6964d17f443dcacfb6a01a0945be4 (127.31.74.195:38549)
I20260806 02:59:43.684552 7376 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.74.195@KRBTEST.COM'} at 127.31.74.195:39609
I20260806 02:59:43.695955 32043 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 02:59:43 dist-test-slave-dhph krb5kdc[7331](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985183, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 02:59:43 dist-test-slave-dhph krb5kdc[7331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:59:43 dist-test-slave-dhph krb5kdc[7331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:59:44.027463 7730 heartbeater.cc:503] Master 127.31.74.254:40379 was elected leader, sending a full tablet report...
I20260806 02:59:44.687968 7873 heartbeater.cc:503] Master 127.31.74.254:40379 was elected leader, sending a full tablet report...
W20260806 02:59:45.032995 7881 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.271s user 0.000s sys 0.024s
W20260806 02:59:45.034309 7758 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:59:43.755958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44450 (local address 127.31.74.254:40379)
0806 02:59:43.756219 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:43.756227 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:43.756524 (+ 297us) server_negotiation.cc:407] Connection header received
0806 02:59:43.756775 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:43.756780 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:43.756853 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:43.756965 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:43.758482 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:43.759307 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:43.760594 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:43.761020 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:43.761158 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:45.033951 (+1272793us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44450: BlockingRecv error: recv got EOF from 127.0.0.1:44450 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Aug 06 02:59:45 dist-test-slave-dhph krb5kdc[7331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985183, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM
I20260806 02:59:45.057439 7758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:45.040249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44456 (local address 127.31.74.254:40379)
0806 02:59:45.040452 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:45.040461 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:45.040891 (+ 430us) server_negotiation.cc:407] Connection header received
0806 02:59:45.041155 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:45.041163 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:45.041240 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:45.041458 (+ 218us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:45.042985 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:45.043782 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:45.045172 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:45.045696 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:45.045850 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:45.050338 (+ 4488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:45.050344 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:45.050348 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:45.050390 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:45.053586 (+ 3196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:45.054526 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:45.054532 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:45.054535 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:45.054797 (+ 262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:45.055474 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:45.055479 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:45.055481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:45.055915 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:45.056611 (+ 696us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:45.057002 (+ 391us) server_negotiation.cc:299] Negotiation successful
0806 02:59:45.057204 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 02:59:45.063180 7376 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44456:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:59:45.071053 7376 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:59:45.099594 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:45.085560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.194:46395 (local address 127.0.0.1:53200)
0806 02:59:45.086910 (+ 1350us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:45.086985 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:45.087259 (+ 274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:45.090779 (+ 3520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:45.090787 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:45.090827 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:59:45.091405 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:45.091426 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:45.093283 (+ 1857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:45.093292 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:45.095843 (+ 2551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:45.095859 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:45.097920 (+ 2061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:45.097926 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:45.098132 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:45.098384 (+ 252us) client_negotiation.cc:769] Sending connection context
0806 02:59:45.098812 (+ 428us) client_negotiation.cc:240] Negotiation successful
0806 02:59:45.099058 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":869,"mutex_wait_us":66,"thread_start_us":659,"threads_started":1}
I20260806 02:59:45.100334 7897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:45.089451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46018 (local address 127.31.74.193:35105)
0806 02:59:45.090012 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:45.090022 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:45.090800 (+ 778us) server_negotiation.cc:407] Connection header received
0806 02:59:45.090929 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:45.090940 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:45.091177 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:45.091385 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:59:45.092411 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:45.094076 (+ 1665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:45.096826 (+ 2750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:45.098344 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:45.098534 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:45.098827 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:45.099894 (+ 1067us) server_negotiation.cc:299] Negotiation successful
0806 02:59:45.100064 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":427,"threads_started":1}
I20260806 02:59:45.100490 7894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:45.085982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53200 (local address 127.31.74.194:46395)
0806 02:59:45.088803 (+ 2821us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:45.088817 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:45.088855 (+ 38us) server_negotiation.cc:407] Connection header received
0806 02:59:45.090159 (+ 1304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:45.090173 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:45.090481 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:45.091663 (+ 1182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:59:45.091901 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:45.093132 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:45.096076 (+ 2944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:45.097683 (+ 1607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:45.097866 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:45.098116 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:45.098984 (+ 868us) server_negotiation.cc:299] Negotiation successful
0806 02:59:45.100092 (+ 1108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2635,"thread_start_us":2436,"threads_started":1}
I20260806 02:59:45.100937 7895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:45.086559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59362 (local address 127.31.74.195:38549)
0806 02:59:45.089153 (+ 2594us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:45.089163 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:45.090189 (+ 1026us) server_negotiation.cc:407] Connection header received
0806 02:59:45.090326 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:45.090336 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:45.090612 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:45.090866 (+ 254us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:59:45.092185 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:45.093977 (+ 1792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:45.097461 (+ 3484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:45.099250 (+ 1789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:45.099428 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:45.099663 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:45.100400 (+ 737us) server_negotiation.cc:299] Negotiation successful
0806 02:59:45.100591 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2408,"thread_start_us":2229,"threads_started":1}
I20260806 02:59:45.101513 7896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:45.089215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:35105 (local address 127.0.0.1:46018)
0806 02:59:45.090481 (+ 1266us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:45.090502 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:45.090671 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:45.091614 (+ 943us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:45.091619 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:45.091647 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:59:45.092183 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:45.092196 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:45.094295 (+ 2099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:45.094299 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:45.096628 (+ 2329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:45.096643 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:45.098881 (+ 2238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:45.098890 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:45.099075 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:45.099111 (+ 36us) client_negotiation.cc:769] Sending connection context
0806 02:59:45.099215 (+ 104us) client_negotiation.cc:240] Negotiation successful
0806 02:59:45.099373 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1133,"mutex_wait_us":4,"thread_start_us":947,"threads_started":1}
I20260806 02:59:45.101545 7893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:45.085853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.195:38549 (local address 127.0.0.1:59362)
0806 02:59:45.089754 (+ 3901us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:45.089784 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:45.089977 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:45.091167 (+ 1190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:45.091172 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:45.091205 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:59:45.091905 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:45.091922 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:45.094152 (+ 2230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:45.094161 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:45.097245 (+ 3084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:45.097261 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:45.099891 (+ 2630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:45.099898 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:45.100140 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:45.100183 (+ 43us) client_negotiation.cc:769] Sending connection context
0806 02:59:45.100307 (+ 124us) client_negotiation.cc:240] Negotiation successful
0806 02:59:45.100472 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3702,"mutex_wait_us":457,"thread_start_us":3755,"threads_started":1}
I20260806 02:59:45.105126 7516 tablet_service.cc:1467] Processing CreateTablet for tablet 31f1f9d1b4bc475cbf5cd4d7e82fda56 (DEFAULT_TABLE table=test-table [id=06f94a6fccf84ec7abbd30388cedf5db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:59:45.105958 7803 tablet_service.cc:1467] Processing CreateTablet for tablet 31f1f9d1b4bc475cbf5cd4d7e82fda56 (DEFAULT_TABLE table=test-table [id=06f94a6fccf84ec7abbd30388cedf5db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:59:45.105494 7662 tablet_service.cc:1467] Processing CreateTablet for tablet 31f1f9d1b4bc475cbf5cd4d7e82fda56 (DEFAULT_TABLE table=test-table [id=06f94a6fccf84ec7abbd30388cedf5db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:59:45.106698 7516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31f1f9d1b4bc475cbf5cd4d7e82fda56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:59:45.107335 7803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31f1f9d1b4bc475cbf5cd4d7e82fda56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:59:45.107744 7662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31f1f9d1b4bc475cbf5cd4d7e82fda56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:59:45.119118 7898 tablet_bootstrap.cc:492] T 31f1f9d1b4bc475cbf5cd4d7e82fda56 P b98ed2c913804afe994cafecc1994347: Bootstrap starting.
I20260806 02:59:45.123714 7898 tablet_bootstrap.cc:654] T 31f1f9d1b4bc475cbf5cd4d7e82fda56 P b98ed2c913804afe994cafecc1994347: Neither blocks nor log segments found. Creating new log.
I20260806 02:59:45.125177 7900 tablet_bootstrap.cc:492] T 31f1f9d1b4bc475cbf5cd4d7e82fda56 P 2fd6a06c3426465484a699dedffbf3b4: Bootstrap starting.
I20260806 02:59:45.125190 7898 log.cc:824] T 31f1f9d1b4bc475cbf5cd4d7e82fda56 P b98ed2c913804afe994cafecc1994347: Log is configured to *not* fsync() on all Append() calls
I20260806 02:59:45.126535 7899 tablet_bootstrap.cc:492] T 31f1f9d1b4bc475cbf5cd4d7e82fda56 P 67c6964d17f443dcacfb6a01a0945be4: Bootstrap starting.
I20260806 02:59:45.129032 7898 tablet_bootstrap.cc:492] T 31f1f9d1b4bc475cbf5cd4d7e82fda56 P b98ed2c913804afe994cafecc1994347: No bootstrap required, opened a new log
I20260806 02:59:45.129383 7898 ts_tablet_manager.cc:1397] T 31f1f9d1b4bc475cbf5cd4d7e82fda56 P b98ed2c913804afe994cafecc1994347: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260806 02:59:45.130581 7900 tablet_bootstrap.cc:654] T 31f1f9d1b4bc475cbf5cd4d7e82fda56 P 2fd6a06c3426465484a699dedffbf3b4: Neither blocks nor log segments found. Creating new log.
I20260806 02:59:45.131191 7899 tablet_bootstrap.cc:654] T 31f1f9d1b4bc475cbf5cd4d7e82fda56 P 67c6964d17f443dcacfb6a01a0945be4: Neither blocks nor log segments found. Creating new log.
I20260806 02:59:45.132027 7900 log.cc:824] T 31f1f9d1b4bc475cbf5cd4d7e82fda56 P 2fd6a06c3426465484a699dedffbf3b4: Log is configured to *not* fsync() on all Append() calls
I20260806 02:59:45.132547 7899 log.cc:824] T 31f1f9d1b4bc475cbf5cd4d7e82fda56 P 67c6964d17f443dcacfb6a01a0945be4: Log is configured to *not* fsync() on all Append() calls
I20260806 02:59:45.135407 7900 tablet_bootstrap.cc:492] T 31f1f9d1b4bc475cbf5cd4d7e82fda56 P 2fd6a06c3426465484a699dedffbf3b4: No bootstrap required, opened a new log
I20260806 02:59:45.135710 7900 ts_tablet_manager.cc:1397] T 31f1f9d1b4bc475cbf5cd4d7e82fda56 P 2fd6a06c3426465484a699dedffbf3b4: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.003s
I20260806 02:59:45.136125 7899 tablet_bootstrap.cc:492] T 31f1f9d1b4bc475cbf5cd4d7e82fda56 P 67c6964d17f443dcacfb6a01a0945be4: No bootstrap required, opened a new log
I20260806 02:59:45.136492 7899 ts_tablet_manager.cc:1397] T 31f1f9d1b4bc475cbf5cd4d7e82fda56 P 67c6964d17f443dcacfb6a01a0945be4: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.002s
I20260806 02:59:45.142649 7898 raft_consensus.cc:348] T 31f1f9d1b4bc475cbf5cd4d7e82fda56 P b98ed2c913804afe994cafecc1994347 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd6a06c3426465484a699dedffbf3b4" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 46395 } } peers { permanent_uuid: "67c6964d17f443dcacfb6a01a0945be4" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 38549 } } peers { permanent_uuid: "b98ed2c913804afe994cafecc1994347" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 35105 } }
I20260806 02:59:45.143177 7898 raft_consensus.cc:374] T 31f1f9d1b4bc475cbf5cd4d7e82fda56 P b98ed2c913804afe994cafecc1994347 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:59:45.143273 7898 raft_consensus.cc:729] T 31f1f9d1b4bc475cbf5cd4d7e82fda56 P b98ed2c913804afe994cafecc1994347 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b98ed2c913804afe994cafecc1994347, State: Initialized, Role: FOLLOWER
I20260806 02:59:45.143989 7898 consensus_queue.cc:260] T 31f1f9d1b4bc475cbf5cd4d7e82fda56 P b98ed2c913804afe994cafecc1994347 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd6a06c3426465484a699dedffbf3b4" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 46395 } } peers { permanent_uuid: "67c6964d17f443dcacfb6a01a0945be4" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 38549 } } peers { permanent_uuid: "b98ed2c913804afe994cafecc1994347" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 35105 } }
I20260806 02:59:45.147228 7898 ts_tablet_manager.cc:1428] T 31f1f9d1b4bc475cbf5cd4d7e82fda56 P b98ed2c913804afe994cafecc1994347: Time spent starting tablet: real 0.018s user 0.017s sys 0.000s
I20260806 02:59:45.147938 7899 raft_consensus.cc:348] T 31f1f9d1b4bc475cbf5cd4d7e82fda56 P 67c6964d17f443dcacfb6a01a0945be4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd6a06c3426465484a699dedffbf3b4" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 46395 } } peers { permanent_uuid: "67c6964d17f443dcacfb6a01a0945be4" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 38549 } } peers { permanent_uuid: "b98ed2c913804afe994cafecc1994347" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 35105 } }
I20260806 02:59:45.147544 7900 raft_consensus.cc:348] T 31f1f9d1b4bc475cbf5cd4d7e82fda56 P 2fd6a06c3426465484a699dedffbf3b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd6a06c3426465484a699dedffbf3b4" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 46395 } } peers { permanent_uuid: "67c6964d17f443dcacfb6a01a0945be4" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 38549 } } peers { permanent_uuid: "b98ed2c913804afe994cafecc1994347" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 35105 } }
I20260806 02:59:45.148356 7899 raft_consensus.cc:374] T 31f1f9d1b4bc475cbf5cd4d7e82fda56 P 67c6964d17f443dcacfb6a01a0945be4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:59:45.148365 7900 raft_consensus.cc:374] T 31f1f9d1b4bc475cbf5cd4d7e82fda56 P 2fd6a06c3426465484a699dedffbf3b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:59:45.148490 7899 raft_consensus.cc:729] T 31f1f9d1b4bc475cbf5cd4d7e82fda56 P 67c6964d17f443dcacfb6a01a0945be4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67c6964d17f443dcacfb6a01a0945be4, State: Initialized, Role: FOLLOWER
I20260806 02:59:45.148488 7900 raft_consensus.cc:729] T 31f1f9d1b4bc475cbf5cd4d7e82fda56 P 2fd6a06c3426465484a699dedffbf3b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fd6a06c3426465484a699dedffbf3b4, State: Initialized, Role: FOLLOWER
I20260806 02:59:45.149130 7899 consensus_queue.cc:260] T 31f1f9d1b4bc475cbf5cd4d7e82fda56 P 67c6964d17f443dcacfb6a01a0945be4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd6a06c3426465484a699dedffbf3b4" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 46395 } } peers { permanent_uuid: "67c6964d17f443dcacfb6a01a0945be4" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 38549 } } peers { permanent_uuid: "b98ed2c913804afe994cafecc1994347" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 35105 } }
I20260806 02:59:45.149343 7900 consensus_queue.cc:260] T 31f1f9d1b4bc475cbf5cd4d7e82fda56 P 2fd6a06c3426465484a699dedffbf3b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd6a06c3426465484a699dedffbf3b4" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 46395 } } peers { permanent_uuid: "67c6964d17f443dcacfb6a01a0945be4" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 38549 } } peers { permanent_uuid: "b98ed2c913804afe994cafecc1994347" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 35105 } }
I20260806 02:59:45.151117 7899 ts_tablet_manager.cc:1428] T 31f1f9d1b4bc475cbf5cd4d7e82fda56 P 67c6964d17f443dcacfb6a01a0945be4: Time spent starting tablet: real 0.014s user 0.013s sys 0.001s
I20260806 02:59:45.155318 7904 raft_consensus.cc:482] T 31f1f9d1b4bc475cbf5cd4d7e82fda56 P b98ed2c913804afe994cafecc1994347 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:59:45.155705 7904 raft_consensus.cc:504] T 31f1f9d1b4bc475cbf5cd4d7e82fda56 P b98ed2c913804afe994cafecc1994347 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd6a06c3426465484a699dedffbf3b4" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 46395 } } peers { permanent_uuid: "67c6964d17f443dcacfb6a01a0945be4" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 38549 } } peers { permanent_uuid: "b98ed2c913804afe994cafecc1994347" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 35105 } }
I20260806 02:59:45.157707 7904 leader_election.cc:288] T 31f1f9d1b4bc475cbf5cd4d7e82fda56 P b98ed2c913804afe994cafecc1994347 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2fd6a06c3426465484a699dedffbf3b4 (127.31.74.194:46395), 67c6964d17f443dcacfb6a01a0945be4 (127.31.74.195:38549)
I20260806 02:59:45.160910 7900 ts_tablet_manager.cc:1428] T 31f1f9d1b4bc475cbf5cd4d7e82fda56 P 2fd6a06c3426465484a699dedffbf3b4: Time spent starting tablet: real 0.025s user 0.016s sys 0.005s
I20260806 02:59:45.169085 7894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:45.159026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.193:53367 (local address 127.31.74.194:46395)
0806 02:59:45.159921 (+ 895us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:45.159929 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:45.162421 (+ 2492us) server_negotiation.cc:407] Connection header received
0806 02:59:45.162794 (+ 373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:45.162799 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:45.162878 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:45.162979 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:59:45.164137 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:45.164910 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:45.167111 (+ 2201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:45.168104 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:45.168296 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:45.168375 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:45.168674 (+ 299us) server_negotiation.cc:299] Negotiation successful
0806 02:59:45.168808 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530}
I20260806 02:59:45.169260 7908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:45.157948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.194:46395 (local address 127.31.74.193:53367)
0806 02:59:45.162298 (+ 4350us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:45.162326 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:45.162522 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:45.163170 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:45.163175 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:45.163219 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:59:45.163979 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:45.163993 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:45.165043 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:45.165050 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:45.166882 (+ 1832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:45.166894 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:45.168323 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:45.168329 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:45.168505 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:45.168530 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 02:59:45.168648 (+ 118us) client_negotiation.cc:240] Negotiation successful
0806 02:59:45.168808 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4154,"spinlock_wait_cycles":19328,"thread_start_us":4988,"threads_started":1}
I20260806 02:59:45.169725 7895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:45.158824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.193:47355 (local address 127.31.74.195:38549)
0806 02:59:45.159506 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:45.159518 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:45.163363 (+ 3845us) server_negotiation.cc:407] Connection header received
0806 02:59:45.163598 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:45.163603 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:45.163687 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:45.163776 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:59:45.164349 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:45.165174 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:45.167360 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:45.168397 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:45.168592 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:45.168686 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:45.169026 (+ 340us) server_negotiation.cc:299] Negotiation successful
0806 02:59:45.169167 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276}
I20260806 02:59:45.170070 7907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:45.158247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.195:38549 (local address 127.31.74.193:47355)
0806 02:59:45.163224 (+ 4977us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:45.163241 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:45.163431 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:45.163814 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:45.163818 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:45.163844 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:59:45.164192 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:45.164201 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:45.165322 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:45.165327 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:45.167148 (+ 1821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:45.167161 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:45.168684 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:45.168692 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:45.168849 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:45.168874 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 02:59:45.169000 (+ 126us) client_negotiation.cc:240] Negotiation successful
0806 02:59:45.169148 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4840,"thread_start_us":3599,"threads_started":1}
I20260806 02:59:45.171507 7685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f1f9d1b4bc475cbf5cd4d7e82fda56" candidate_uuid: "b98ed2c913804afe994cafecc1994347" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fd6a06c3426465484a699dedffbf3b4" is_pre_election: true
I20260806 02:59:45.171505 7828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f1f9d1b4bc475cbf5cd4d7e82fda56" candidate_uuid: "b98ed2c913804afe994cafecc1994347" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67c6964d17f443dcacfb6a01a0945be4" is_pre_election: true
I20260806 02:59:45.172112 7828 raft_consensus.cc:2454] T 31f1f9d1b4bc475cbf5cd4d7e82fda56 P 67c6964d17f443dcacfb6a01a0945be4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b98ed2c913804afe994cafecc1994347 in term 0.
I20260806 02:59:45.172106 7685 raft_consensus.cc:2454] T 31f1f9d1b4bc475cbf5cd4d7e82fda56 P 2fd6a06c3426465484a699dedffbf3b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b98ed2c913804afe994cafecc1994347 in term 0.
I20260806 02:59:45.172829 7465 leader_election.cc:302] T 31f1f9d1b4bc475cbf5cd4d7e82fda56 P b98ed2c913804afe994cafecc1994347 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67c6964d17f443dcacfb6a01a0945be4, b98ed2c913804afe994cafecc1994347; no voters:
I20260806 02:59:45.173308 7904 raft_consensus.cc:2781] T 31f1f9d1b4bc475cbf5cd4d7e82fda56 P b98ed2c913804afe994cafecc1994347 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:59:45.173527 7904 raft_consensus.cc:482] T 31f1f9d1b4bc475cbf5cd4d7e82fda56 P b98ed2c913804afe994cafecc1994347 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:59:45.173656 7904 raft_consensus.cc:3037] T 31f1f9d1b4bc475cbf5cd4d7e82fda56 P b98ed2c913804afe994cafecc1994347 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:59:45.176532 7904 raft_consensus.cc:504] T 31f1f9d1b4bc475cbf5cd4d7e82fda56 P b98ed2c913804afe994cafecc1994347 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd6a06c3426465484a699dedffbf3b4" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 46395 } } peers { permanent_uuid: "67c6964d17f443dcacfb6a01a0945be4" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 38549 } } peers { permanent_uuid: "b98ed2c913804afe994cafecc1994347" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 35105 } }
I20260806 02:59:45.177539 7904 leader_election.cc:288] T 31f1f9d1b4bc475cbf5cd4d7e82fda56 P b98ed2c913804afe994cafecc1994347 [CANDIDATE]: Term 1 election: Requested vote from peers 2fd6a06c3426465484a699dedffbf3b4 (127.31.74.194:46395), 67c6964d17f443dcacfb6a01a0945be4 (127.31.74.195:38549)
W20260806 02:59:45.178478 7874 tablet.cc:2369] T 31f1f9d1b4bc475cbf5cd4d7e82fda56 P 67c6964d17f443dcacfb6a01a0945be4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:59:45.178961 7828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f1f9d1b4bc475cbf5cd4d7e82fda56" candidate_uuid: "b98ed2c913804afe994cafecc1994347" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67c6964d17f443dcacfb6a01a0945be4"
I20260806 02:59:45.179273 7828 raft_consensus.cc:3037] T 31f1f9d1b4bc475cbf5cd4d7e82fda56 P 67c6964d17f443dcacfb6a01a0945be4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:59:45.182400 7828 raft_consensus.cc:2454] T 31f1f9d1b4bc475cbf5cd4d7e82fda56 P 67c6964d17f443dcacfb6a01a0945be4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b98ed2c913804afe994cafecc1994347 in term 1.
I20260806 02:59:45.182969 7685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f1f9d1b4bc475cbf5cd4d7e82fda56" candidate_uuid: "b98ed2c913804afe994cafecc1994347" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fd6a06c3426465484a699dedffbf3b4"
I20260806 02:59:45.183267 7685 raft_consensus.cc:3037] T 31f1f9d1b4bc475cbf5cd4d7e82fda56 P 2fd6a06c3426465484a699dedffbf3b4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:59:45.183269 7465 leader_election.cc:302] T 31f1f9d1b4bc475cbf5cd4d7e82fda56 P b98ed2c913804afe994cafecc1994347 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67c6964d17f443dcacfb6a01a0945be4, b98ed2c913804afe994cafecc1994347; no voters:
I20260806 02:59:45.183828 7904 raft_consensus.cc:2781] T 31f1f9d1b4bc475cbf5cd4d7e82fda56 P b98ed2c913804afe994cafecc1994347 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:59:45.184703 7904 raft_consensus.cc:686] T 31f1f9d1b4bc475cbf5cd4d7e82fda56 P b98ed2c913804afe994cafecc1994347 [term 1 LEADER]: Becoming Leader. State: Replica: b98ed2c913804afe994cafecc1994347, State: Running, Role: LEADER
I20260806 02:59:45.185392 7904 consensus_queue.cc:237] T 31f1f9d1b4bc475cbf5cd4d7e82fda56 P b98ed2c913804afe994cafecc1994347 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd6a06c3426465484a699dedffbf3b4" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 46395 } } peers { permanent_uuid: "67c6964d17f443dcacfb6a01a0945be4" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 38549 } } peers { permanent_uuid: "b98ed2c913804afe994cafecc1994347" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 35105 } }
I20260806 02:59:45.185837 7685 raft_consensus.cc:2454] T 31f1f9d1b4bc475cbf5cd4d7e82fda56 P 2fd6a06c3426465484a699dedffbf3b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b98ed2c913804afe994cafecc1994347 in term 1.
I20260806 02:59:45.190985 7376 catalog_manager.cc:5495] T 31f1f9d1b4bc475cbf5cd4d7e82fda56 P b98ed2c913804afe994cafecc1994347 reported cstate change: term changed from 0 to 1, leader changed from <none> to b98ed2c913804afe994cafecc1994347 (127.31.74.193). New cstate: current_term: 1 leader_uuid: "b98ed2c913804afe994cafecc1994347" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd6a06c3426465484a699dedffbf3b4" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 46395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67c6964d17f443dcacfb6a01a0945be4" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 38549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b98ed2c913804afe994cafecc1994347" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 35105 } health_report { overall_health: HEALTHY } } }
I20260806 02:59:45.230787 7897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:45.224103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46034 (local address 127.31.74.193:35105)
0806 02:59:45.224360 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:45.224365 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:45.224490 (+ 125us) server_negotiation.cc:407] Connection header received
0806 02:59:45.224736 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:45.224741 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:45.224838 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:45.224977 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:59:45.225476 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:45.226251 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:45.227822 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:45.228384 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:45.228686 (+ 302us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:45.229385 (+ 699us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:59:45.229971 (+ 586us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:59:45.230090 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:45.230287 (+ 197us) server_negotiation.cc:299] Negotiation successful
0806 02:59:45.230373 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260806 02:59:45.239647 7828 raft_consensus.cc:1261] T 31f1f9d1b4bc475cbf5cd4d7e82fda56 P 67c6964d17f443dcacfb6a01a0945be4 [term 1 FOLLOWER]: Refusing update from remote peer b98ed2c913804afe994cafecc1994347: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:59:45.239648 7685 raft_consensus.cc:1261] T 31f1f9d1b4bc475cbf5cd4d7e82fda56 P 2fd6a06c3426465484a699dedffbf3b4 [term 1 FOLLOWER]: Refusing update from remote peer b98ed2c913804afe994cafecc1994347: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:59:45.240645 7909 consensus_queue.cc:1035] T 31f1f9d1b4bc475cbf5cd4d7e82fda56 P b98ed2c913804afe994cafecc1994347 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fd6a06c3426465484a699dedffbf3b4" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 46395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:59:45.241088 7904 consensus_queue.cc:1035] T 31f1f9d1b4bc475cbf5cd4d7e82fda56 P b98ed2c913804afe994cafecc1994347 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67c6964d17f443dcacfb6a01a0945be4" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 38549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:59:45.253310 7913 mvcc.cc:205] Tried to move back new op lower bound from 7315395318730481664 to 7315395318525620224. Current Snapshot: MvccSnapshot[applied={T|T < 7315395318730481664}]
I20260806 02:59:45.254834 7914 mvcc.cc:205] Tried to move back new op lower bound from 7315395318730481664 to 7315395318525620224. Current Snapshot: MvccSnapshot[applied={T|T < 7315395318730481664}]
I20260806 02:59:45.270715 7375 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44456:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:59:45.271023 7375 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44456:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:59:45.274771 7375 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8643cb88550544428f8c7eee4349597e: Sending DeleteTablet for 3 replicas of tablet 31f1f9d1b4bc475cbf5cd4d7e82fda56
I20260806 02:59:45.276055 7662 tablet_service.cc:1514] Processing DeleteTablet for tablet 31f1f9d1b4bc475cbf5cd4d7e82fda56 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:59:45 UTC) from {username='oryx', principal='oryx/127.31.74.254@KRBTEST.COM'} at 127.0.0.1:53200
I20260806 02:59:45.276093 7803 tablet_service.cc:1514] Processing DeleteTablet for tablet 31f1f9d1b4bc475cbf5cd4d7e82fda56 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:59:45 UTC) from {username='oryx', principal='oryx/127.31.74.254@KRBTEST.COM'} at 127.0.0.1:59362
I20260806 02:59:45.277133 7924 tablet_replica.cc:330] T 31f1f9d1b4bc475cbf5cd4d7e82fda56 P 67c6964d17f443dcacfb6a01a0945be4: stopping tablet replica
I20260806 02:59:45.277946 7924 raft_consensus.cc:2229] T 31f1f9d1b4bc475cbf5cd4d7e82fda56 P 67c6964d17f443dcacfb6a01a0945be4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:59:45.278254 7516 tablet_service.cc:1514] Processing DeleteTablet for tablet 31f1f9d1b4bc475cbf5cd4d7e82fda56 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:59:45 UTC) from {username='oryx', principal='oryx/127.31.74.254@KRBTEST.COM'} at 127.0.0.1:46018
I20260806 02:59:45.278633 7924 raft_consensus.cc:2258] T 31f1f9d1b4bc475cbf5cd4d7e82fda56 P 67c6964d17f443dcacfb6a01a0945be4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:59:45.280812 7375 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44456:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260806 02:59:45.281100 7924 ts_tablet_manager.cc:1905] T 31f1f9d1b4bc475cbf5cd4d7e82fda56 P 67c6964d17f443dcacfb6a01a0945be4: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260806 02:59:45.281550 7375 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:59:45.287125 7925 tablet_replica.cc:330] T 31f1f9d1b4bc475cbf5cd4d7e82fda56 P 2fd6a06c3426465484a699dedffbf3b4: stopping tablet replica
I20260806 02:59:45.287631 7925 raft_consensus.cc:2229] T 31f1f9d1b4bc475cbf5cd4d7e82fda56 P 2fd6a06c3426465484a699dedffbf3b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:59:45.289644 7924 ts_tablet_manager.cc:1918] T 31f1f9d1b4bc475cbf5cd4d7e82fda56 P 67c6964d17f443dcacfb6a01a0945be4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:59:45.289901 7924 log.cc:1196] T 31f1f9d1b4bc475cbf5cd4d7e82fda56 P 67c6964d17f443dcacfb6a01a0945be4: Deleting WAL directory at /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/minicluster-data/ts-2/wal/wals/31f1f9d1b4bc475cbf5cd4d7e82fda56
I20260806 02:59:45.290045 7925 raft_consensus.cc:2258] T 31f1f9d1b4bc475cbf5cd4d7e82fda56 P 2fd6a06c3426465484a699dedffbf3b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:59:45.290657 7924 ts_tablet_manager.cc:1939] T 31f1f9d1b4bc475cbf5cd4d7e82fda56 P 67c6964d17f443dcacfb6a01a0945be4: Deleting consensus metadata
I20260806 02:59:45.291803 7515 tablet_service.cc:1467] Processing CreateTablet for tablet 10f01425a71041159c1a13b410937c4a (DEFAULT_TABLE table=test-table [id=b16ddab0a4444dcb9fb2cec29309581a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:59:45.292472 7515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10f01425a71041159c1a13b410937c4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:59:45.292713 7360 catalog_manager.cc:4841] TS 67c6964d17f443dcacfb6a01a0945be4 (127.31.74.195:38549): tablet 31f1f9d1b4bc475cbf5cd4d7e82fda56 (table test-table [id=06f94a6fccf84ec7abbd30388cedf5db]) successfully deleted
I20260806 02:59:45.295773 7803 tablet_service.cc:1467] Processing CreateTablet for tablet 10f01425a71041159c1a13b410937c4a (DEFAULT_TABLE table=test-table [id=b16ddab0a4444dcb9fb2cec29309581a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:59:45.296308 7803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10f01425a71041159c1a13b410937c4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:59:45.296820 7925 ts_tablet_manager.cc:1905] T 31f1f9d1b4bc475cbf5cd4d7e82fda56 P 2fd6a06c3426465484a699dedffbf3b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:59:45.299804 7898 tablet_bootstrap.cc:492] T 10f01425a71041159c1a13b410937c4a P b98ed2c913804afe994cafecc1994347: Bootstrap starting.
I20260806 02:59:45.301715 7898 tablet_bootstrap.cc:654] T 10f01425a71041159c1a13b410937c4a P b98ed2c913804afe994cafecc1994347: Neither blocks nor log segments found. Creating new log.
I20260806 02:59:45.306120 7925 ts_tablet_manager.cc:1918] T 31f1f9d1b4bc475cbf5cd4d7e82fda56 P 2fd6a06c3426465484a699dedffbf3b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:59:45.306488 7925 log.cc:1196] T 31f1f9d1b4bc475cbf5cd4d7e82fda56 P 2fd6a06c3426465484a699dedffbf3b4: Deleting WAL directory at /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/minicluster-data/ts-1/wal/wals/31f1f9d1b4bc475cbf5cd4d7e82fda56
I20260806 02:59:45.307407 7925 ts_tablet_manager.cc:1939] T 31f1f9d1b4bc475cbf5cd4d7e82fda56 P 2fd6a06c3426465484a699dedffbf3b4: Deleting consensus metadata
I20260806 02:59:45.309191 7899 tablet_bootstrap.cc:492] T 10f01425a71041159c1a13b410937c4a P 67c6964d17f443dcacfb6a01a0945be4: Bootstrap starting.
I20260806 02:59:45.309978 7363 catalog_manager.cc:4841] TS 2fd6a06c3426465484a699dedffbf3b4 (127.31.74.194:46395): tablet 31f1f9d1b4bc475cbf5cd4d7e82fda56 (table test-table [id=06f94a6fccf84ec7abbd30388cedf5db]) successfully deleted
I20260806 02:59:45.310995 7899 tablet_bootstrap.cc:654] T 10f01425a71041159c1a13b410937c4a P 67c6964d17f443dcacfb6a01a0945be4: Neither blocks nor log segments found. Creating new log.
I20260806 02:59:45.312461 7898 tablet_bootstrap.cc:492] T 10f01425a71041159c1a13b410937c4a P b98ed2c913804afe994cafecc1994347: No bootstrap required, opened a new log
I20260806 02:59:45.312724 7898 ts_tablet_manager.cc:1397] T 10f01425a71041159c1a13b410937c4a P b98ed2c913804afe994cafecc1994347: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.003s
I20260806 02:59:45.313019 7662 tablet_service.cc:1467] Processing CreateTablet for tablet 10f01425a71041159c1a13b410937c4a (DEFAULT_TABLE table=test-table [id=b16ddab0a4444dcb9fb2cec29309581a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:59:45.313505 7899 tablet_bootstrap.cc:492] T 10f01425a71041159c1a13b410937c4a P 67c6964d17f443dcacfb6a01a0945be4: No bootstrap required, opened a new log
I20260806 02:59:45.313659 7899 ts_tablet_manager.cc:1397] T 10f01425a71041159c1a13b410937c4a P 67c6964d17f443dcacfb6a01a0945be4: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260806 02:59:45.313936 7898 raft_consensus.cc:348] T 10f01425a71041159c1a13b410937c4a P b98ed2c913804afe994cafecc1994347 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98ed2c913804afe994cafecc1994347" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 35105 } } peers { permanent_uuid: "2fd6a06c3426465484a699dedffbf3b4" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 46395 } } peers { permanent_uuid: "67c6964d17f443dcacfb6a01a0945be4" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 38549 } }
I20260806 02:59:45.314256 7898 raft_consensus.cc:374] T 10f01425a71041159c1a13b410937c4a P b98ed2c913804afe994cafecc1994347 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:59:45.314282 7662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10f01425a71041159c1a13b410937c4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:59:45.314437 7898 raft_consensus.cc:729] T 10f01425a71041159c1a13b410937c4a P b98ed2c913804afe994cafecc1994347 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b98ed2c913804afe994cafecc1994347, State: Initialized, Role: FOLLOWER
I20260806 02:59:45.314505 7899 raft_consensus.cc:348] T 10f01425a71041159c1a13b410937c4a P 67c6964d17f443dcacfb6a01a0945be4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98ed2c913804afe994cafecc1994347" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 35105 } } peers { permanent_uuid: "2fd6a06c3426465484a699dedffbf3b4" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 46395 } } peers { permanent_uuid: "67c6964d17f443dcacfb6a01a0945be4" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 38549 } }
I20260806 02:59:45.314760 7899 raft_consensus.cc:374] T 10f01425a71041159c1a13b410937c4a P 67c6964d17f443dcacfb6a01a0945be4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:59:45.314867 7899 raft_consensus.cc:729] T 10f01425a71041159c1a13b410937c4a P 67c6964d17f443dcacfb6a01a0945be4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67c6964d17f443dcacfb6a01a0945be4, State: Initialized, Role: FOLLOWER
I20260806 02:59:45.315150 7899 consensus_queue.cc:260] T 10f01425a71041159c1a13b410937c4a P 67c6964d17f443dcacfb6a01a0945be4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98ed2c913804afe994cafecc1994347" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 35105 } } peers { permanent_uuid: "2fd6a06c3426465484a699dedffbf3b4" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 46395 } } peers { permanent_uuid: "67c6964d17f443dcacfb6a01a0945be4" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 38549 } }
I20260806 02:59:45.315186 7898 consensus_queue.cc:260] T 10f01425a71041159c1a13b410937c4a P b98ed2c913804afe994cafecc1994347 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98ed2c913804afe994cafecc1994347" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 35105 } } peers { permanent_uuid: "2fd6a06c3426465484a699dedffbf3b4" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 46395 } } peers { permanent_uuid: "67c6964d17f443dcacfb6a01a0945be4" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 38549 } }
I20260806 02:59:45.315802 7899 ts_tablet_manager.cc:1428] T 10f01425a71041159c1a13b410937c4a P 67c6964d17f443dcacfb6a01a0945be4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260806 02:59:45.318483 7898 ts_tablet_manager.cc:1428] T 10f01425a71041159c1a13b410937c4a P b98ed2c913804afe994cafecc1994347: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260806 02:59:45.319870 7926 tablet_replica.cc:330] T 31f1f9d1b4bc475cbf5cd4d7e82fda56 P b98ed2c913804afe994cafecc1994347: stopping tablet replica
I20260806 02:59:45.320942 7926 raft_consensus.cc:2229] T 31f1f9d1b4bc475cbf5cd4d7e82fda56 P b98ed2c913804afe994cafecc1994347 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:59:45.321935 7926 raft_consensus.cc:2258] T 31f1f9d1b4bc475cbf5cd4d7e82fda56 P b98ed2c913804afe994cafecc1994347 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:59:45.323828 7900 tablet_bootstrap.cc:492] T 10f01425a71041159c1a13b410937c4a P 2fd6a06c3426465484a699dedffbf3b4: Bootstrap starting.
I20260806 02:59:45.324592 7926 ts_tablet_manager.cc:1905] T 31f1f9d1b4bc475cbf5cd4d7e82fda56 P b98ed2c913804afe994cafecc1994347: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:59:45.325666 7900 tablet_bootstrap.cc:654] T 10f01425a71041159c1a13b410937c4a P 2fd6a06c3426465484a699dedffbf3b4: Neither blocks nor log segments found. Creating new log.
I20260806 02:59:45.328235 7900 tablet_bootstrap.cc:492] T 10f01425a71041159c1a13b410937c4a P 2fd6a06c3426465484a699dedffbf3b4: No bootstrap required, opened a new log
I20260806 02:59:45.328414 7900 ts_tablet_manager.cc:1397] T 10f01425a71041159c1a13b410937c4a P 2fd6a06c3426465484a699dedffbf3b4: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20260806 02:59:45.329088 7900 raft_consensus.cc:348] T 10f01425a71041159c1a13b410937c4a P 2fd6a06c3426465484a699dedffbf3b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98ed2c913804afe994cafecc1994347" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 35105 } } peers { permanent_uuid: "2fd6a06c3426465484a699dedffbf3b4" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 46395 } } peers { permanent_uuid: "67c6964d17f443dcacfb6a01a0945be4" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 38549 } }
I20260806 02:59:45.329424 7900 raft_consensus.cc:374] T 10f01425a71041159c1a13b410937c4a P 2fd6a06c3426465484a699dedffbf3b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:59:45.329562 7900 raft_consensus.cc:729] T 10f01425a71041159c1a13b410937c4a P 2fd6a06c3426465484a699dedffbf3b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fd6a06c3426465484a699dedffbf3b4, State: Initialized, Role: FOLLOWER
I20260806 02:59:45.329883 7900 consensus_queue.cc:260] T 10f01425a71041159c1a13b410937c4a P 2fd6a06c3426465484a699dedffbf3b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98ed2c913804afe994cafecc1994347" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 35105 } } peers { permanent_uuid: "2fd6a06c3426465484a699dedffbf3b4" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 46395 } } peers { permanent_uuid: "67c6964d17f443dcacfb6a01a0945be4" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 38549 } }
I20260806 02:59:45.330513 7900 ts_tablet_manager.cc:1428] T 10f01425a71041159c1a13b410937c4a P 2fd6a06c3426465484a699dedffbf3b4: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260806 02:59:45.333463 7926 ts_tablet_manager.cc:1918] T 31f1f9d1b4bc475cbf5cd4d7e82fda56 P b98ed2c913804afe994cafecc1994347: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:59:45.333669 7926 log.cc:1196] T 31f1f9d1b4bc475cbf5cd4d7e82fda56 P b98ed2c913804afe994cafecc1994347: Deleting WAL directory at /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/minicluster-data/ts-0/wal/wals/31f1f9d1b4bc475cbf5cd4d7e82fda56
I20260806 02:59:45.334268 7926 ts_tablet_manager.cc:1939] T 31f1f9d1b4bc475cbf5cd4d7e82fda56 P b98ed2c913804afe994cafecc1994347: Deleting consensus metadata
I20260806 02:59:45.336300 7362 catalog_manager.cc:4841] TS b98ed2c913804afe994cafecc1994347 (127.31.74.193:35105): tablet 31f1f9d1b4bc475cbf5cd4d7e82fda56 (table test-table [id=06f94a6fccf84ec7abbd30388cedf5db]) successfully deleted
W20260806 02:59:45.366856 7589 tablet.cc:2369] T 10f01425a71041159c1a13b410937c4a P b98ed2c913804afe994cafecc1994347: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:59:45.522497 7731 tablet.cc:2369] T 10f01425a71041159c1a13b410937c4a P 2fd6a06c3426465484a699dedffbf3b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:59:45.674980 7922 raft_consensus.cc:482] T 10f01425a71041159c1a13b410937c4a P b98ed2c913804afe994cafecc1994347 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:59:45.675215 7922 raft_consensus.cc:504] T 10f01425a71041159c1a13b410937c4a P b98ed2c913804afe994cafecc1994347 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98ed2c913804afe994cafecc1994347" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 35105 } } peers { permanent_uuid: "2fd6a06c3426465484a699dedffbf3b4" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 46395 } } peers { permanent_uuid: "67c6964d17f443dcacfb6a01a0945be4" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 38549 } }
I20260806 02:59:45.676154 7922 leader_election.cc:288] T 10f01425a71041159c1a13b410937c4a P b98ed2c913804afe994cafecc1994347 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2fd6a06c3426465484a699dedffbf3b4 (127.31.74.194:46395), 67c6964d17f443dcacfb6a01a0945be4 (127.31.74.195:38549)
I20260806 02:59:45.677316 7906 raft_consensus.cc:482] T 10f01425a71041159c1a13b410937c4a P 2fd6a06c3426465484a699dedffbf3b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:59:45.677258 7828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10f01425a71041159c1a13b410937c4a" candidate_uuid: "b98ed2c913804afe994cafecc1994347" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67c6964d17f443dcacfb6a01a0945be4" is_pre_election: true
I20260806 02:59:45.677513 7906 raft_consensus.cc:504] T 10f01425a71041159c1a13b410937c4a P 2fd6a06c3426465484a699dedffbf3b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98ed2c913804afe994cafecc1994347" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 35105 } } peers { permanent_uuid: "2fd6a06c3426465484a699dedffbf3b4" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 46395 } } peers { permanent_uuid: "67c6964d17f443dcacfb6a01a0945be4" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 38549 } }
I20260806 02:59:45.677697 7828 raft_consensus.cc:2454] T 10f01425a71041159c1a13b410937c4a P 67c6964d17f443dcacfb6a01a0945be4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b98ed2c913804afe994cafecc1994347 in term 0.
I20260806 02:59:45.678817 7465 leader_election.cc:302] T 10f01425a71041159c1a13b410937c4a P b98ed2c913804afe994cafecc1994347 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67c6964d17f443dcacfb6a01a0945be4, b98ed2c913804afe994cafecc1994347; no voters:
I20260806 02:59:45.678789 7685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10f01425a71041159c1a13b410937c4a" candidate_uuid: "b98ed2c913804afe994cafecc1994347" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fd6a06c3426465484a699dedffbf3b4" is_pre_election: true
I20260806 02:59:45.679005 7685 raft_consensus.cc:2454] T 10f01425a71041159c1a13b410937c4a P 2fd6a06c3426465484a699dedffbf3b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b98ed2c913804afe994cafecc1994347 in term 0.
I20260806 02:59:45.679284 7922 raft_consensus.cc:2781] T 10f01425a71041159c1a13b410937c4a P b98ed2c913804afe994cafecc1994347 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:59:45.679440 7922 raft_consensus.cc:482] T 10f01425a71041159c1a13b410937c4a P b98ed2c913804afe994cafecc1994347 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:59:45.679540 7922 raft_consensus.cc:3037] T 10f01425a71041159c1a13b410937c4a P b98ed2c913804afe994cafecc1994347 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:59:45.680639 7906 leader_election.cc:288] T 10f01425a71041159c1a13b410937c4a P 2fd6a06c3426465484a699dedffbf3b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b98ed2c913804afe994cafecc1994347 (127.31.74.193:35105), 67c6964d17f443dcacfb6a01a0945be4 (127.31.74.195:38549)
I20260806 02:59:45.683455 7922 raft_consensus.cc:504] T 10f01425a71041159c1a13b410937c4a P b98ed2c913804afe994cafecc1994347 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98ed2c913804afe994cafecc1994347" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 35105 } } peers { permanent_uuid: "2fd6a06c3426465484a699dedffbf3b4" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 46395 } } peers { permanent_uuid: "67c6964d17f443dcacfb6a01a0945be4" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 38549 } }
I20260806 02:59:45.684892 7685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10f01425a71041159c1a13b410937c4a" candidate_uuid: "b98ed2c913804afe994cafecc1994347" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fd6a06c3426465484a699dedffbf3b4"
I20260806 02:59:45.685199 7685 raft_consensus.cc:3037] T 10f01425a71041159c1a13b410937c4a P 2fd6a06c3426465484a699dedffbf3b4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:59:45.688879 7685 raft_consensus.cc:2454] T 10f01425a71041159c1a13b410937c4a P 2fd6a06c3426465484a699dedffbf3b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b98ed2c913804afe994cafecc1994347 in term 1.
I20260806 02:59:45.689579 7468 leader_election.cc:302] T 10f01425a71041159c1a13b410937c4a P b98ed2c913804afe994cafecc1994347 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fd6a06c3426465484a699dedffbf3b4, b98ed2c913804afe994cafecc1994347; no voters:
I20260806 02:59:45.689989 7904 raft_consensus.cc:2781] T 10f01425a71041159c1a13b410937c4a P b98ed2c913804afe994cafecc1994347 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:59:45.690166 7904 raft_consensus.cc:686] T 10f01425a71041159c1a13b410937c4a P b98ed2c913804afe994cafecc1994347 [term 1 LEADER]: Becoming Leader. State: Replica: b98ed2c913804afe994cafecc1994347, State: Running, Role: LEADER
I20260806 02:59:45.690490 7904 consensus_queue.cc:237] T 10f01425a71041159c1a13b410937c4a P b98ed2c913804afe994cafecc1994347 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98ed2c913804afe994cafecc1994347" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 35105 } } peers { permanent_uuid: "2fd6a06c3426465484a699dedffbf3b4" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 46395 } } peers { permanent_uuid: "67c6964d17f443dcacfb6a01a0945be4" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 38549 } }
I20260806 02:59:45.694257 7828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10f01425a71041159c1a13b410937c4a" candidate_uuid: "b98ed2c913804afe994cafecc1994347" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67c6964d17f443dcacfb6a01a0945be4"
I20260806 02:59:45.694554 7828 raft_consensus.cc:3037] T 10f01425a71041159c1a13b410937c4a P 67c6964d17f443dcacfb6a01a0945be4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:59:45.698087 7828 raft_consensus.cc:2454] T 10f01425a71041159c1a13b410937c4a P 67c6964d17f443dcacfb6a01a0945be4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b98ed2c913804afe994cafecc1994347 in term 1.
I20260806 02:59:45.699038 7922 leader_election.cc:288] T 10f01425a71041159c1a13b410937c4a P b98ed2c913804afe994cafecc1994347 [CANDIDATE]: Term 1 election: Requested vote from peers 2fd6a06c3426465484a699dedffbf3b4 (127.31.74.194:46395), 67c6964d17f443dcacfb6a01a0945be4 (127.31.74.195:38549)
I20260806 02:59:45.708389 7376 catalog_manager.cc:5495] T 10f01425a71041159c1a13b410937c4a P b98ed2c913804afe994cafecc1994347 reported cstate change: term changed from 0 to 1, leader changed from <none> to b98ed2c913804afe994cafecc1994347 (127.31.74.193). New cstate: current_term: 1 leader_uuid: "b98ed2c913804afe994cafecc1994347" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98ed2c913804afe994cafecc1994347" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 35105 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2fd6a06c3426465484a699dedffbf3b4" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 46395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67c6964d17f443dcacfb6a01a0945be4" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 38549 } health_report { overall_health: UNKNOWN } } }
I20260806 02:59:45.709427 7897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:45.681990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.194:54515 (local address 127.31.74.193:35105)
0806 02:59:45.682222 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:45.682229 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:45.700503 (+ 18274us) server_negotiation.cc:407] Connection header received
0806 02:59:45.700956 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:45.700963 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:45.701043 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:45.701117 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:59:45.701904 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:45.703549 (+ 1645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:45.706453 (+ 2904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:45.708064 (+ 1611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:45.708218 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:45.708305 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:45.708870 (+ 565us) server_negotiation.cc:299] Negotiation successful
0806 02:59:45.709039 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 02:59:45.711851 7932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:45.681065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.195:38549 (local address 127.31.74.194:53409)
0806 02:59:45.698062 (+ 16997us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:45.698092 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:45.698281 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:45.704433 (+ 6152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:45.704438 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:45.704468 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:59:45.704925 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:45.704940 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:45.706667 (+ 1727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:45.706673 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:45.708857 (+ 2184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:45.708870 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:45.710868 (+ 1998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:45.710875 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:45.711076 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:45.711116 (+ 40us) client_negotiation.cc:769] Sending connection context
0806 02:59:45.711222 (+ 106us) client_negotiation.cc:240] Negotiation successful
0806 02:59:45.711374 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16798,"mutex_wait_us":4,"thread_start_us":19458,"threads_started":1}
I20260806 02:59:45.712704 7931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:45.681460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.194:53409 (local address 127.31.74.195:38549)
0806 02:59:45.703884 (+ 22424us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:45.703893 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:45.703913 (+ 20us) server_negotiation.cc:407] Connection header received
0806 02:59:45.703988 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:45.703993 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:45.704076 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:45.704183 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:59:45.705218 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:45.706464 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:45.709032 (+ 2568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:45.710399 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:45.710605 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:45.710691 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:45.712085 (+ 1394us) server_negotiation.cc:299] Negotiation successful
0806 02:59:45.712269 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22270,"thread_start_us":23598,"threads_started":1}
I20260806 02:59:45.713570 7828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10f01425a71041159c1a13b410937c4a" candidate_uuid: "2fd6a06c3426465484a699dedffbf3b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67c6964d17f443dcacfb6a01a0945be4" is_pre_election: true
I20260806 02:59:45.713955 7828 raft_consensus.cc:2379] T 10f01425a71041159c1a13b410937c4a P 67c6964d17f443dcacfb6a01a0945be4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2fd6a06c3426465484a699dedffbf3b4 in current term 1: Already voted for candidate b98ed2c913804afe994cafecc1994347 in this term.
I20260806 02:59:45.714463 7930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:45.682030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:35105 (local address 127.31.74.194:54515)
0806 02:59:45.700377 (+ 18347us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:45.700400 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:45.700774 (+ 374us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:45.701361 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:45.701368 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:45.701397 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:59:45.701705 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:45.701715 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:45.703807 (+ 2092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:45.703813 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:45.706207 (+ 2394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:45.706223 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:45.708478 (+ 2255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:45.708485 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:45.708656 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:45.708691 (+ 35us) client_negotiation.cc:769] Sending connection context
0806 02:59:45.713833 (+ 5142us) client_negotiation.cc:240] Negotiation successful
0806 02:59:45.714017 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18058,"thread_start_us":15625,"threads_started":1}
I20260806 02:59:45.715561 7537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10f01425a71041159c1a13b410937c4a" candidate_uuid: "2fd6a06c3426465484a699dedffbf3b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b98ed2c913804afe994cafecc1994347" is_pre_election: true
I20260806 02:59:45.716693 7610 leader_election.cc:302] T 10f01425a71041159c1a13b410937c4a P 2fd6a06c3426465484a699dedffbf3b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2fd6a06c3426465484a699dedffbf3b4; no voters: 67c6964d17f443dcacfb6a01a0945be4, b98ed2c913804afe994cafecc1994347
I20260806 02:59:45.717127 7906 raft_consensus.cc:2726] T 10f01425a71041159c1a13b410937c4a P 2fd6a06c3426465484a699dedffbf3b4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
Aug 06 02:59:45 dist-test-slave-dhph krb5kdc[7331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985181, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.74.254@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM
I20260806 02:59:45.808490 7939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:45.793364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:40379 (local address 127.0.0.1:44466)
0806 02:59:45.794223 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:45.794241 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:45.794468 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:45.794947 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:45.794951 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:45.795650 (+ 699us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:59:45.795918 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:45.795929 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:45.797124 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:45.797129 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:45.798187 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:45.798200 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:45.799100 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:45.799104 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:45.799242 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:45.799981 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:59:45.800005 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:59:45.802447 (+ 2442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:59:45.805752 (+ 3305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:45.805764 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:45.805770 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:45.806260 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:45.806765 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:45.806769 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:45.806772 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:45.806922 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:45.807514 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:59:45.807521 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:59:45.807700 (+ 179us) client_negotiation.cc:769] Sending connection context
0806 02:59:45.807907 (+ 207us) client_negotiation.cc:240] Negotiation successful
0806 02:59:45.808139 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":720,"thread_start_us":591,"threads_started":1}
I20260806 02:59:45.808672 7940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:45.793406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44466 (local address 127.31.74.254:40379)
0806 02:59:45.794125 (+ 719us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:45.794131 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:45.794425 (+ 294us) server_negotiation.cc:407] Connection header received
0806 02:59:45.794683 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:45.794688 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:45.794755 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:45.794851 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:45.796092 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:45.796972 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:45.798359 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:45.798903 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:45.799016 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:45.802735 (+ 3719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:45.802740 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:45.802749 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:45.802781 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:45.805599 (+ 2818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:45.806393 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:45.806397 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:45.806399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:45.806526 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:45.807035 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:45.807039 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:45.807041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:45.807348 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:45.807455 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:45.808092 (+ 637us) server_negotiation.cc:299] Negotiation successful
0806 02:59:45.808274 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":313,"threads_started":1}
I20260806 02:59:45.814162 7376 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.31.74.254@KRBTEST.COM'} at 127.0.0.1:44466:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260806 02:59:45.815357 7376 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:59:45.823539 7516 tablet_service.cc:1467] Processing CreateTablet for tablet 2d7b42182581448c901a552ffa35f182 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e6c0f074eb054de6a6e40037f03b559f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 02:59:45.824010 7662 tablet_service.cc:1467] Processing CreateTablet for tablet 2d7b42182581448c901a552ffa35f182 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e6c0f074eb054de6a6e40037f03b559f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 02:59:45.824185 7516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d7b42182581448c901a552ffa35f182. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:59:45.824007 7803 tablet_service.cc:1467] Processing CreateTablet for tablet 2d7b42182581448c901a552ffa35f182 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e6c0f074eb054de6a6e40037f03b559f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 02:59:45.824483 7662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d7b42182581448c901a552ffa35f182. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:59:45.824508 7803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d7b42182581448c901a552ffa35f182. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:59:45.835407 7942 tablet_bootstrap.cc:492] T 2d7b42182581448c901a552ffa35f182 P 67c6964d17f443dcacfb6a01a0945be4: Bootstrap starting.
I20260806 02:59:45.836380 7943 tablet_bootstrap.cc:492] T 2d7b42182581448c901a552ffa35f182 P b98ed2c913804afe994cafecc1994347: Bootstrap starting.
I20260806 02:59:45.838498 7943 tablet_bootstrap.cc:654] T 2d7b42182581448c901a552ffa35f182 P b98ed2c913804afe994cafecc1994347: Neither blocks nor log segments found. Creating new log.
I20260806 02:59:45.838642 7942 tablet_bootstrap.cc:654] T 2d7b42182581448c901a552ffa35f182 P 67c6964d17f443dcacfb6a01a0945be4: Neither blocks nor log segments found. Creating new log.
I20260806 02:59:45.839425 7941 tablet_bootstrap.cc:492] T 2d7b42182581448c901a552ffa35f182 P 2fd6a06c3426465484a699dedffbf3b4: Bootstrap starting.
I20260806 02:59:45.841320 7941 tablet_bootstrap.cc:654] T 2d7b42182581448c901a552ffa35f182 P 2fd6a06c3426465484a699dedffbf3b4: Neither blocks nor log segments found. Creating new log.
I20260806 02:59:45.841611 7942 tablet_bootstrap.cc:492] T 2d7b42182581448c901a552ffa35f182 P 67c6964d17f443dcacfb6a01a0945be4: No bootstrap required, opened a new log
I20260806 02:59:45.841837 7942 ts_tablet_manager.cc:1397] T 2d7b42182581448c901a552ffa35f182 P 67c6964d17f443dcacfb6a01a0945be4: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260806 02:59:45.843021 7942 raft_consensus.cc:348] T 2d7b42182581448c901a552ffa35f182 P 67c6964d17f443dcacfb6a01a0945be4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98ed2c913804afe994cafecc1994347" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 35105 } } peers { permanent_uuid: "2fd6a06c3426465484a699dedffbf3b4" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 46395 } } peers { permanent_uuid: "67c6964d17f443dcacfb6a01a0945be4" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 38549 } }
I20260806 02:59:45.843408 7942 raft_consensus.cc:374] T 2d7b42182581448c901a552ffa35f182 P 67c6964d17f443dcacfb6a01a0945be4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:59:45.843536 7942 raft_consensus.cc:729] T 2d7b42182581448c901a552ffa35f182 P 67c6964d17f443dcacfb6a01a0945be4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67c6964d17f443dcacfb6a01a0945be4, State: Initialized, Role: FOLLOWER
I20260806 02:59:45.844134 7942 consensus_queue.cc:260] T 2d7b42182581448c901a552ffa35f182 P 67c6964d17f443dcacfb6a01a0945be4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98ed2c913804afe994cafecc1994347" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 35105 } } peers { permanent_uuid: "2fd6a06c3426465484a699dedffbf3b4" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 46395 } } peers { permanent_uuid: "67c6964d17f443dcacfb6a01a0945be4" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 38549 } }
I20260806 02:59:45.845129 7941 tablet_bootstrap.cc:492] T 2d7b42182581448c901a552ffa35f182 P 2fd6a06c3426465484a699dedffbf3b4: No bootstrap required, opened a new log
I20260806 02:59:45.845363 7941 ts_tablet_manager.cc:1397] T 2d7b42182581448c901a552ffa35f182 P 2fd6a06c3426465484a699dedffbf3b4: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260806 02:59:45.845395 7942 ts_tablet_manager.cc:1428] T 2d7b42182581448c901a552ffa35f182 P 67c6964d17f443dcacfb6a01a0945be4: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260806 02:59:45.846048 7947 tablet_replica.cc:439] T 2d7b42182581448c901a552ffa35f182 P 67c6964d17f443dcacfb6a01a0945be4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98ed2c913804afe994cafecc1994347" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 35105 } } peers { permanent_uuid: "2fd6a06c3426465484a699dedffbf3b4" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 46395 } } peers { permanent_uuid: "67c6964d17f443dcacfb6a01a0945be4" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 38549 } } }
I20260806 02:59:45.846321 7941 raft_consensus.cc:348] T 2d7b42182581448c901a552ffa35f182 P 2fd6a06c3426465484a699dedffbf3b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98ed2c913804afe994cafecc1994347" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 35105 } } peers { permanent_uuid: "2fd6a06c3426465484a699dedffbf3b4" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 46395 } } peers { permanent_uuid: "67c6964d17f443dcacfb6a01a0945be4" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 38549 } }
I20260806 02:59:45.846652 7947 tablet_replica.cc:442] T 2d7b42182581448c901a552ffa35f182 P 67c6964d17f443dcacfb6a01a0945be4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:59:45.846951 7941 raft_consensus.cc:374] T 2d7b42182581448c901a552ffa35f182 P 2fd6a06c3426465484a699dedffbf3b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:59:45.847237 7943 tablet_bootstrap.cc:492] T 2d7b42182581448c901a552ffa35f182 P b98ed2c913804afe994cafecc1994347: No bootstrap required, opened a new log
I20260806 02:59:45.847095 7941 raft_consensus.cc:729] T 2d7b42182581448c901a552ffa35f182 P 2fd6a06c3426465484a699dedffbf3b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fd6a06c3426465484a699dedffbf3b4, State: Initialized, Role: FOLLOWER
I20260806 02:59:45.847432 7943 ts_tablet_manager.cc:1397] T 2d7b42182581448c901a552ffa35f182 P b98ed2c913804afe994cafecc1994347: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260806 02:59:45.847674 7941 consensus_queue.cc:260] T 2d7b42182581448c901a552ffa35f182 P 2fd6a06c3426465484a699dedffbf3b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98ed2c913804afe994cafecc1994347" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 35105 } } peers { permanent_uuid: "2fd6a06c3426465484a699dedffbf3b4" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 46395 } } peers { permanent_uuid: "67c6964d17f443dcacfb6a01a0945be4" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 38549 } }
I20260806 02:59:45.848228 7906 tablet_replica.cc:439] T 2d7b42182581448c901a552ffa35f182 P 2fd6a06c3426465484a699dedffbf3b4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98ed2c913804afe994cafecc1994347" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 35105 } } peers { permanent_uuid: "2fd6a06c3426465484a699dedffbf3b4" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 46395 } } peers { permanent_uuid: "67c6964d17f443dcacfb6a01a0945be4" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 38549 } } }
I20260806 02:59:45.848383 7943 raft_consensus.cc:348] T 2d7b42182581448c901a552ffa35f182 P b98ed2c913804afe994cafecc1994347 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98ed2c913804afe994cafecc1994347" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 35105 } } peers { permanent_uuid: "2fd6a06c3426465484a699dedffbf3b4" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 46395 } } peers { permanent_uuid: "67c6964d17f443dcacfb6a01a0945be4" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 38549 } }
I20260806 02:59:45.848537 7906 tablet_replica.cc:442] T 2d7b42182581448c901a552ffa35f182 P 2fd6a06c3426465484a699dedffbf3b4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:59:45.848635 7943 raft_consensus.cc:374] T 2d7b42182581448c901a552ffa35f182 P b98ed2c913804afe994cafecc1994347 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:59:45.848745 7943 raft_consensus.cc:729] T 2d7b42182581448c901a552ffa35f182 P b98ed2c913804afe994cafecc1994347 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b98ed2c913804afe994cafecc1994347, State: Initialized, Role: FOLLOWER
I20260806 02:59:45.849080 7941 ts_tablet_manager.cc:1428] T 2d7b42182581448c901a552ffa35f182 P 2fd6a06c3426465484a699dedffbf3b4: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260806 02:59:45.849000 7943 consensus_queue.cc:260] T 2d7b42182581448c901a552ffa35f182 P b98ed2c913804afe994cafecc1994347 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98ed2c913804afe994cafecc1994347" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 35105 } } peers { permanent_uuid: "2fd6a06c3426465484a699dedffbf3b4" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 46395 } } peers { permanent_uuid: "67c6964d17f443dcacfb6a01a0945be4" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 38549 } }
I20260806 02:59:45.849704 7943 ts_tablet_manager.cc:1428] T 2d7b42182581448c901a552ffa35f182 P b98ed2c913804afe994cafecc1994347: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260806 02:59:45.849615 7904 tablet_replica.cc:439] T 2d7b42182581448c901a552ffa35f182 P b98ed2c913804afe994cafecc1994347: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98ed2c913804afe994cafecc1994347" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 35105 } } peers { permanent_uuid: "2fd6a06c3426465484a699dedffbf3b4" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 46395 } } peers { permanent_uuid: "67c6964d17f443dcacfb6a01a0945be4" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 38549 } } }
I20260806 02:59:45.849968 7904 tablet_replica.cc:442] T 2d7b42182581448c901a552ffa35f182 P b98ed2c913804afe994cafecc1994347: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:59:45.870819 7904 raft_consensus.cc:482] T 2d7b42182581448c901a552ffa35f182 P b98ed2c913804afe994cafecc1994347 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:59:45.871068 7904 raft_consensus.cc:504] T 2d7b42182581448c901a552ffa35f182 P b98ed2c913804afe994cafecc1994347 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98ed2c913804afe994cafecc1994347" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 35105 } } peers { permanent_uuid: "2fd6a06c3426465484a699dedffbf3b4" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 46395 } } peers { permanent_uuid: "67c6964d17f443dcacfb6a01a0945be4" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 38549 } }
I20260806 02:59:45.871886 7904 leader_election.cc:288] T 2d7b42182581448c901a552ffa35f182 P b98ed2c913804afe994cafecc1994347 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2fd6a06c3426465484a699dedffbf3b4 (127.31.74.194:46395), 67c6964d17f443dcacfb6a01a0945be4 (127.31.74.195:38549)
I20260806 02:59:45.872457 7685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7b42182581448c901a552ffa35f182" candidate_uuid: "b98ed2c913804afe994cafecc1994347" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fd6a06c3426465484a699dedffbf3b4" is_pre_election: true
I20260806 02:59:45.872542 7828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7b42182581448c901a552ffa35f182" candidate_uuid: "b98ed2c913804afe994cafecc1994347" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67c6964d17f443dcacfb6a01a0945be4" is_pre_election: true
I20260806 02:59:45.872789 7685 raft_consensus.cc:2454] T 2d7b42182581448c901a552ffa35f182 P 2fd6a06c3426465484a699dedffbf3b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b98ed2c913804afe994cafecc1994347 in term 0.
I20260806 02:59:45.872840 7828 raft_consensus.cc:2454] T 2d7b42182581448c901a552ffa35f182 P 67c6964d17f443dcacfb6a01a0945be4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b98ed2c913804afe994cafecc1994347 in term 0.
I20260806 02:59:45.873337 7465 leader_election.cc:302] T 2d7b42182581448c901a552ffa35f182 P b98ed2c913804afe994cafecc1994347 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67c6964d17f443dcacfb6a01a0945be4, b98ed2c913804afe994cafecc1994347; no voters:
I20260806 02:59:45.873647 7904 raft_consensus.cc:2781] T 2d7b42182581448c901a552ffa35f182 P b98ed2c913804afe994cafecc1994347 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:59:45.873747 7904 raft_consensus.cc:482] T 2d7b42182581448c901a552ffa35f182 P b98ed2c913804afe994cafecc1994347 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:59:45.873854 7904 raft_consensus.cc:3037] T 2d7b42182581448c901a552ffa35f182 P b98ed2c913804afe994cafecc1994347 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:59:45.876145 7904 raft_consensus.cc:504] T 2d7b42182581448c901a552ffa35f182 P b98ed2c913804afe994cafecc1994347 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98ed2c913804afe994cafecc1994347" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 35105 } } peers { permanent_uuid: "2fd6a06c3426465484a699dedffbf3b4" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 46395 } } peers { permanent_uuid: "67c6964d17f443dcacfb6a01a0945be4" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 38549 } }
I20260806 02:59:45.876899 7904 leader_election.cc:288] T 2d7b42182581448c901a552ffa35f182 P b98ed2c913804afe994cafecc1994347 [CANDIDATE]: Term 1 election: Requested vote from peers 2fd6a06c3426465484a699dedffbf3b4 (127.31.74.194:46395), 67c6964d17f443dcacfb6a01a0945be4 (127.31.74.195:38549)
I20260806 02:59:45.877481 7828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7b42182581448c901a552ffa35f182" candidate_uuid: "b98ed2c913804afe994cafecc1994347" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67c6964d17f443dcacfb6a01a0945be4"
I20260806 02:59:45.877755 7828 raft_consensus.cc:3037] T 2d7b42182581448c901a552ffa35f182 P 67c6964d17f443dcacfb6a01a0945be4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:59:45.877480 7685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7b42182581448c901a552ffa35f182" candidate_uuid: "b98ed2c913804afe994cafecc1994347" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fd6a06c3426465484a699dedffbf3b4"
I20260806 02:59:45.878754 7685 raft_consensus.cc:3037] T 2d7b42182581448c901a552ffa35f182 P 2fd6a06c3426465484a699dedffbf3b4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:59:45.880435 7828 raft_consensus.cc:2454] T 2d7b42182581448c901a552ffa35f182 P 67c6964d17f443dcacfb6a01a0945be4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b98ed2c913804afe994cafecc1994347 in term 1.
I20260806 02:59:45.880975 7465 leader_election.cc:302] T 2d7b42182581448c901a552ffa35f182 P b98ed2c913804afe994cafecc1994347 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67c6964d17f443dcacfb6a01a0945be4, b98ed2c913804afe994cafecc1994347; no voters:
I20260806 02:59:45.881304 7904 raft_consensus.cc:2781] T 2d7b42182581448c901a552ffa35f182 P b98ed2c913804afe994cafecc1994347 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:59:45.881510 7904 raft_consensus.cc:686] T 2d7b42182581448c901a552ffa35f182 P b98ed2c913804afe994cafecc1994347 [term 1 LEADER]: Becoming Leader. State: Replica: b98ed2c913804afe994cafecc1994347, State: Running, Role: LEADER
I20260806 02:59:45.881865 7685 raft_consensus.cc:2454] T 2d7b42182581448c901a552ffa35f182 P 2fd6a06c3426465484a699dedffbf3b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b98ed2c913804afe994cafecc1994347 in term 1.
I20260806 02:59:45.881835 7904 consensus_queue.cc:237] T 2d7b42182581448c901a552ffa35f182 P b98ed2c913804afe994cafecc1994347 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98ed2c913804afe994cafecc1994347" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 35105 } } peers { permanent_uuid: "2fd6a06c3426465484a699dedffbf3b4" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 46395 } } peers { permanent_uuid: "67c6964d17f443dcacfb6a01a0945be4" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 38549 } }
I20260806 02:59:45.883544 7922 tablet_replica.cc:439] T 2d7b42182581448c901a552ffa35f182 P b98ed2c913804afe994cafecc1994347: TxnStatusTablet state changed. Reason: New leader b98ed2c913804afe994cafecc1994347. Latest consensus state: current_term: 1 leader_uuid: "b98ed2c913804afe994cafecc1994347" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98ed2c913804afe994cafecc1994347" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 35105 } } peers { permanent_uuid: "2fd6a06c3426465484a699dedffbf3b4" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 46395 } } peers { permanent_uuid: "67c6964d17f443dcacfb6a01a0945be4" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 38549 } } }
I20260806 02:59:45.883834 7922 tablet_replica.cc:442] T 2d7b42182581448c901a552ffa35f182 P b98ed2c913804afe994cafecc1994347: This TxnStatusTablet replica's current role is: LEADER
I20260806 02:59:45.884534 7949 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260806 02:59:45.884781 7949 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260806 02:59:45.885116 7375 catalog_manager.cc:5495] T 2d7b42182581448c901a552ffa35f182 P b98ed2c913804afe994cafecc1994347 reported cstate change: term changed from 0 to 1, leader changed from <none> to b98ed2c913804afe994cafecc1994347 (127.31.74.193). New cstate: current_term: 1 leader_uuid: "b98ed2c913804afe994cafecc1994347" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98ed2c913804afe994cafecc1994347" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 35105 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2fd6a06c3426465484a699dedffbf3b4" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 46395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67c6964d17f443dcacfb6a01a0945be4" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 38549 } health_report { overall_health: UNKNOWN } } }
I20260806 02:59:45.909438 7897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:45.901729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46040 (local address 127.31.74.193:35105)
0806 02:59:45.902006 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:45.902012 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:45.902187 (+ 175us) server_negotiation.cc:407] Connection header received
0806 02:59:45.903484 (+ 1297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:45.903490 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:45.903590 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:45.903689 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:59:45.904441 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:45.905343 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:45.907398 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:45.907894 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:45.908014 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:45.908680 (+ 666us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:59:45.908810 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:59:45.908893 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:45.909097 (+ 204us) server_negotiation.cc:299] Negotiation successful
0806 02:59:45.909178 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260806 02:59:45.909638 7939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:45.901733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:35105 (local address 127.0.0.1:46040)
0806 02:59:45.902037 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:45.902062 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:45.902256 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:45.903765 (+ 1509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:45.903774 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:45.903792 (+ 18us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 02:59:45.904028 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:45.904042 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:45.905506 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:45.905514 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:45.907192 (+ 1678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:45.907213 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:45.908086 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:45.908104 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:45.908362 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:45.908455 (+ 93us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 02:59:45.908931 (+ 476us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 02:59:45.908954 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 02:59:45.909064 (+ 110us) client_negotiation.cc:240] Negotiation successful
0806 02:59:45.909259 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260806 02:59:45.914654 7828 raft_consensus.cc:1261] T 2d7b42182581448c901a552ffa35f182 P 67c6964d17f443dcacfb6a01a0945be4 [term 1 FOLLOWER]: Refusing update from remote peer b98ed2c913804afe994cafecc1994347: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:59:45.914608 7685 raft_consensus.cc:1261] T 2d7b42182581448c901a552ffa35f182 P 2fd6a06c3426465484a699dedffbf3b4 [term 1 FOLLOWER]: Refusing update from remote peer b98ed2c913804afe994cafecc1994347: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:59:45.915426 7922 consensus_queue.cc:1035] T 2d7b42182581448c901a552ffa35f182 P b98ed2c913804afe994cafecc1994347 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fd6a06c3426465484a699dedffbf3b4" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 46395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:59:45.915750 7904 consensus_queue.cc:1035] T 2d7b42182581448c901a552ffa35f182 P b98ed2c913804afe994cafecc1994347 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67c6964d17f443dcacfb6a01a0945be4" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 38549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:59:45.922824 7906 tablet_replica.cc:439] T 2d7b42182581448c901a552ffa35f182 P 2fd6a06c3426465484a699dedffbf3b4: TxnStatusTablet state changed. Reason: New leader b98ed2c913804afe994cafecc1994347. Latest consensus state: current_term: 1 leader_uuid: "b98ed2c913804afe994cafecc1994347" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98ed2c913804afe994cafecc1994347" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 35105 } } peers { permanent_uuid: "2fd6a06c3426465484a699dedffbf3b4" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 46395 } } peers { permanent_uuid: "67c6964d17f443dcacfb6a01a0945be4" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 38549 } } }
I20260806 02:59:45.923238 7906 tablet_replica.cc:442] T 2d7b42182581448c901a552ffa35f182 P 2fd6a06c3426465484a699dedffbf3b4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:59:45.926179 7918 tablet_replica.cc:439] T 2d7b42182581448c901a552ffa35f182 P b98ed2c913804afe994cafecc1994347: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b98ed2c913804afe994cafecc1994347" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98ed2c913804afe994cafecc1994347" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 35105 } } peers { permanent_uuid: "2fd6a06c3426465484a699dedffbf3b4" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 46395 } } peers { permanent_uuid: "67c6964d17f443dcacfb6a01a0945be4" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 38549 } } }
I20260806 02:59:45.926687 7918 tablet_replica.cc:442] T 2d7b42182581448c901a552ffa35f182 P b98ed2c913804afe994cafecc1994347: This TxnStatusTablet replica's current role is: LEADER
I20260806 02:59:45.936435 7947 tablet_replica.cc:439] T 2d7b42182581448c901a552ffa35f182 P 67c6964d17f443dcacfb6a01a0945be4: TxnStatusTablet state changed. Reason: New leader b98ed2c913804afe994cafecc1994347. Latest consensus state: current_term: 1 leader_uuid: "b98ed2c913804afe994cafecc1994347" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98ed2c913804afe994cafecc1994347" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 35105 } } peers { permanent_uuid: "2fd6a06c3426465484a699dedffbf3b4" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 46395 } } peers { permanent_uuid: "67c6964d17f443dcacfb6a01a0945be4" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 38549 } } }
I20260806 02:59:45.937111 7947 tablet_replica.cc:442] T 2d7b42182581448c901a552ffa35f182 P 67c6964d17f443dcacfb6a01a0945be4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:59:45.941323 7922 tablet_replica.cc:439] T 2d7b42182581448c901a552ffa35f182 P b98ed2c913804afe994cafecc1994347: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b98ed2c913804afe994cafecc1994347" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98ed2c913804afe994cafecc1994347" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 35105 } } peers { permanent_uuid: "2fd6a06c3426465484a699dedffbf3b4" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 46395 } } peers { permanent_uuid: "67c6964d17f443dcacfb6a01a0945be4" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 38549 } } }
I20260806 02:59:45.941699 7922 tablet_replica.cc:442] T 2d7b42182581448c901a552ffa35f182 P b98ed2c913804afe994cafecc1994347: This TxnStatusTablet replica's current role is: LEADER
I20260806 02:59:45.943082 7906 tablet_replica.cc:439] T 2d7b42182581448c901a552ffa35f182 P 2fd6a06c3426465484a699dedffbf3b4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b98ed2c913804afe994cafecc1994347" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98ed2c913804afe994cafecc1994347" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 35105 } } peers { permanent_uuid: "2fd6a06c3426465484a699dedffbf3b4" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 46395 } } peers { permanent_uuid: "67c6964d17f443dcacfb6a01a0945be4" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 38549 } } }
I20260806 02:59:45.943475 7906 tablet_replica.cc:442] T 2d7b42182581448c901a552ffa35f182 P 2fd6a06c3426465484a699dedffbf3b4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:59:45.949199 7947 tablet_replica.cc:439] T 2d7b42182581448c901a552ffa35f182 P 67c6964d17f443dcacfb6a01a0945be4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b98ed2c913804afe994cafecc1994347" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98ed2c913804afe994cafecc1994347" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 35105 } } peers { permanent_uuid: "2fd6a06c3426465484a699dedffbf3b4" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 46395 } } peers { permanent_uuid: "67c6964d17f443dcacfb6a01a0945be4" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 38549 } } }
I20260806 02:59:45.949543 7947 tablet_replica.cc:442] T 2d7b42182581448c901a552ffa35f182 P 67c6964d17f443dcacfb6a01a0945be4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:59:45.964354 7897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:45.956479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.193:44053 (local address 127.31.74.193:35105)
0806 02:59:45.956734 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:45.956741 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:45.957448 (+ 707us) server_negotiation.cc:407] Connection header received
0806 02:59:45.957622 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:45.957627 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:45.957717 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:45.957837 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:59:45.958494 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:45.959823 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:45.961638 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:45.962433 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:45.962618 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:45.963214 (+ 596us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:59:45.963520 (+ 306us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:59:45.963631 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:45.963816 (+ 185us) server_negotiation.cc:299] Negotiation successful
0806 02:59:45.963917 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 02:59:45.964370 7962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:45.956505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:35105 (local address 127.31.74.193:44053)
0806 02:59:45.957313 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:45.957333 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:45.957475 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:45.957896 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:45.957900 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:45.957911 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 02:59:45.958260 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:45.958269 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:45.960038 (+ 1769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:45.960043 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:45.961460 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:45.961474 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:45.962725 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:45.962731 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:45.962902 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:45.962995 (+ 93us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 02:59:45.963655 (+ 660us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 02:59:45.963666 (+ 11us) client_negotiation.cc:769] Sending connection context
0806 02:59:45.963744 (+ 78us) client_negotiation.cc:240] Negotiation successful
0806 02:59:45.963918 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"mutex_wait_us":4,"thread_start_us":456,"threads_started":1}
I20260806 02:59:45.975514 7828 raft_consensus.cc:1261] T 10f01425a71041159c1a13b410937c4a P 67c6964d17f443dcacfb6a01a0945be4 [term 1 FOLLOWER]: Refusing update from remote peer b98ed2c913804afe994cafecc1994347: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:59:45.975745 7685 raft_consensus.cc:1261] T 10f01425a71041159c1a13b410937c4a P 2fd6a06c3426465484a699dedffbf3b4 [term 1 FOLLOWER]: Refusing update from remote peer b98ed2c913804afe994cafecc1994347: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:59:45.976181 7918 consensus_queue.cc:1035] T 10f01425a71041159c1a13b410937c4a P b98ed2c913804afe994cafecc1994347 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67c6964d17f443dcacfb6a01a0945be4" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 38549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:59:45.976581 7904 consensus_queue.cc:1035] T 10f01425a71041159c1a13b410937c4a P b98ed2c913804afe994cafecc1994347 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fd6a06c3426465484a699dedffbf3b4" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 46395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:59:45.987668 7954 mvcc.cc:205] Tried to move back new op lower bound from 7315395321749970944 to 7315395320591613952. Current Snapshot: MvccSnapshot[applied={T|T < 7315395321749970944}]
I20260806 02:59:46.059536 7375 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44456:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:59:46.059840 7375 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44456:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:59:46.064034 7375 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8643cb88550544428f8c7eee4349597e: Sending DeleteTablet for 3 replicas of tablet 10f01425a71041159c1a13b410937c4a
I20260806 02:59:46.065436 7662 tablet_service.cc:1514] Processing DeleteTablet for tablet 10f01425a71041159c1a13b410937c4a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:59:46 UTC) from {username='oryx', principal='oryx/127.31.74.254@KRBTEST.COM'} at 127.0.0.1:53200
I20260806 02:59:46.066228 7966 tablet_replica.cc:330] T 10f01425a71041159c1a13b410937c4a P 2fd6a06c3426465484a699dedffbf3b4: stopping tablet replica
I20260806 02:59:46.066491 7803 tablet_service.cc:1514] Processing DeleteTablet for tablet 10f01425a71041159c1a13b410937c4a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:59:46 UTC) from {username='oryx', principal='oryx/127.31.74.254@KRBTEST.COM'} at 127.0.0.1:59362
I20260806 02:59:46.066547 7966 raft_consensus.cc:2229] T 10f01425a71041159c1a13b410937c4a P 2fd6a06c3426465484a699dedffbf3b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:59:46.066946 7966 raft_consensus.cc:2258] T 10f01425a71041159c1a13b410937c4a P 2fd6a06c3426465484a699dedffbf3b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:59:46.075718 7516 tablet_service.cc:1514] Processing DeleteTablet for tablet 10f01425a71041159c1a13b410937c4a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:59:46 UTC) from {username='oryx', principal='oryx/127.31.74.254@KRBTEST.COM'} at 127.0.0.1:46018
I20260806 02:59:46.076813 7968 tablet_replica.cc:330] T 10f01425a71041159c1a13b410937c4a P b98ed2c913804afe994cafecc1994347: stopping tablet replica
I20260806 02:59:46.077195 7968 raft_consensus.cc:2229] T 10f01425a71041159c1a13b410937c4a P b98ed2c913804afe994cafecc1994347 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:59:46.077922 7968 raft_consensus.cc:2258] T 10f01425a71041159c1a13b410937c4a P b98ed2c913804afe994cafecc1994347 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:59:46.079694 7968 ts_tablet_manager.cc:1905] T 10f01425a71041159c1a13b410937c4a P b98ed2c913804afe994cafecc1994347: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:59:46.080018 7966 ts_tablet_manager.cc:1905] T 10f01425a71041159c1a13b410937c4a P 2fd6a06c3426465484a699dedffbf3b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:59:46.089113 7966 ts_tablet_manager.cc:1918] T 10f01425a71041159c1a13b410937c4a P 2fd6a06c3426465484a699dedffbf3b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:59:46.089402 7966 log.cc:1196] T 10f01425a71041159c1a13b410937c4a P 2fd6a06c3426465484a699dedffbf3b4: Deleting WAL directory at /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/minicluster-data/ts-1/wal/wals/10f01425a71041159c1a13b410937c4a
I20260806 02:59:46.090001 7966 ts_tablet_manager.cc:1939] T 10f01425a71041159c1a13b410937c4a P 2fd6a06c3426465484a699dedffbf3b4: Deleting consensus metadata
I20260806 02:59:46.091364 7363 catalog_manager.cc:4841] TS 2fd6a06c3426465484a699dedffbf3b4 (127.31.74.194:46395): tablet 10f01425a71041159c1a13b410937c4a (table test-table [id=b16ddab0a4444dcb9fb2cec29309581a]) successfully deleted
I20260806 02:59:46.091835 7968 ts_tablet_manager.cc:1918] T 10f01425a71041159c1a13b410937c4a P b98ed2c913804afe994cafecc1994347: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:59:46.092103 7968 log.cc:1196] T 10f01425a71041159c1a13b410937c4a P b98ed2c913804afe994cafecc1994347: Deleting WAL directory at /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/minicluster-data/ts-0/wal/wals/10f01425a71041159c1a13b410937c4a
I20260806 02:59:46.092846 7968 ts_tablet_manager.cc:1939] T 10f01425a71041159c1a13b410937c4a P b98ed2c913804afe994cafecc1994347: Deleting consensus metadata
I20260806 02:59:46.094574 7362 catalog_manager.cc:4841] TS b98ed2c913804afe994cafecc1994347 (127.31.74.193:35105): tablet 10f01425a71041159c1a13b410937c4a (table test-table [id=b16ddab0a4444dcb9fb2cec29309581a]) successfully deleted
I20260806 02:59:46.095067 7967 tablet_replica.cc:330] T 10f01425a71041159c1a13b410937c4a P 67c6964d17f443dcacfb6a01a0945be4: stopping tablet replica
I20260806 02:59:46.095328 7967 raft_consensus.cc:2229] T 10f01425a71041159c1a13b410937c4a P 67c6964d17f443dcacfb6a01a0945be4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:59:46.095630 7967 raft_consensus.cc:2258] T 10f01425a71041159c1a13b410937c4a P 67c6964d17f443dcacfb6a01a0945be4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:59:46.096707 7967 ts_tablet_manager.cc:1905] T 10f01425a71041159c1a13b410937c4a P 67c6964d17f443dcacfb6a01a0945be4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:59:46.103058 7967 ts_tablet_manager.cc:1918] T 10f01425a71041159c1a13b410937c4a P 67c6964d17f443dcacfb6a01a0945be4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:59:46.103251 7967 log.cc:1196] T 10f01425a71041159c1a13b410937c4a P 67c6964d17f443dcacfb6a01a0945be4: Deleting WAL directory at /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/minicluster-data/ts-2/wal/wals/10f01425a71041159c1a13b410937c4a
I20260806 02:59:46.103744 7967 ts_tablet_manager.cc:1939] T 10f01425a71041159c1a13b410937c4a P 67c6964d17f443dcacfb6a01a0945be4: Deleting consensus metadata
I20260806 02:59:46.105014 7360 catalog_manager.cc:4841] TS 67c6964d17f443dcacfb6a01a0945be4 (127.31.74.195:38549): tablet 10f01425a71041159c1a13b410937c4a (table test-table [id=b16ddab0a4444dcb9fb2cec29309581a]) successfully deleted
Aug 06 02:59:46 dist-test-slave-dhph krb5kdc[7331](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 02:59:46 dist-test-slave-dhph krb5kdc[7331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:59:46 dist-test-slave-dhph krb5kdc[7331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
W20260806 02:59:46.152704 7940 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:59:46.139944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44476 (local address 127.31.74.254:40379)
0806 02:59:46.140160 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:46.140167 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:46.140687 (+ 520us) server_negotiation.cc:407] Connection header received
0806 02:59:46.140871 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:46.140876 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:46.140978 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:46.141105 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:46.142799 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:46.143923 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:46.145176 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:46.145827 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:46.145958 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:46.152388 (+ 6430us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44476: BlockingRecv error: recv got EOF from 127.0.0.1:44476 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Aug 06 02:59:46 dist-test-slave-dhph krb5kdc[7331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM
I20260806 02:59:46.175668 7940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:46.156354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44480 (local address 127.31.74.254:40379)
0806 02:59:46.156571 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:46.156578 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:46.156917 (+ 339us) server_negotiation.cc:407] Connection header received
0806 02:59:46.157094 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:46.157100 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:46.157193 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:46.157342 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:46.158835 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:46.160113 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:46.162050 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:46.162831 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:46.162978 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:46.167460 (+ 4482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:46.167468 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:46.167474 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:46.167560 (+ 86us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:46.171874 (+ 4314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:46.172975 (+ 1101us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:46.172983 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:46.173082 (+ 99us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:46.173320 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:46.173768 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:46.173774 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:46.173778 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:46.174212 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:46.174508 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:46.174957 (+ 449us) server_negotiation.cc:299] Negotiation successful
0806 02:59:46.175220 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 02:59:46.180105 7375 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44480:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:59:46.180768 7375 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:59:46.188736 7516 tablet_service.cc:1467] Processing CreateTablet for tablet 3c679f4befa14b1d8b08c3813c7a89e8 (DEFAULT_TABLE table=test-table [id=d834d902cce54ca495abb2918dd25429]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:59:46.189256 7516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c679f4befa14b1d8b08c3813c7a89e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:59:46.189234 7662 tablet_service.cc:1467] Processing CreateTablet for tablet 3c679f4befa14b1d8b08c3813c7a89e8 (DEFAULT_TABLE table=test-table [id=d834d902cce54ca495abb2918dd25429]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:59:46.189705 7662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c679f4befa14b1d8b08c3813c7a89e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:59:46.194345 7943 tablet_bootstrap.cc:492] T 3c679f4befa14b1d8b08c3813c7a89e8 P b98ed2c913804afe994cafecc1994347: Bootstrap starting.
I20260806 02:59:46.195814 7943 tablet_bootstrap.cc:654] T 3c679f4befa14b1d8b08c3813c7a89e8 P b98ed2c913804afe994cafecc1994347: Neither blocks nor log segments found. Creating new log.
I20260806 02:59:46.197124 7803 tablet_service.cc:1467] Processing CreateTablet for tablet 3c679f4befa14b1d8b08c3813c7a89e8 (DEFAULT_TABLE table=test-table [id=d834d902cce54ca495abb2918dd25429]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:59:46.197610 7803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c679f4befa14b1d8b08c3813c7a89e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:59:46.198484 7943 tablet_bootstrap.cc:492] T 3c679f4befa14b1d8b08c3813c7a89e8 P b98ed2c913804afe994cafecc1994347: No bootstrap required, opened a new log
I20260806 02:59:46.198693 7943 ts_tablet_manager.cc:1397] T 3c679f4befa14b1d8b08c3813c7a89e8 P b98ed2c913804afe994cafecc1994347: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260806 02:59:46.199385 7943 raft_consensus.cc:348] T 3c679f4befa14b1d8b08c3813c7a89e8 P b98ed2c913804afe994cafecc1994347 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98ed2c913804afe994cafecc1994347" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 35105 } } peers { permanent_uuid: "2fd6a06c3426465484a699dedffbf3b4" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 46395 } } peers { permanent_uuid: "67c6964d17f443dcacfb6a01a0945be4" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 38549 } }
I20260806 02:59:46.199658 7943 raft_consensus.cc:374] T 3c679f4befa14b1d8b08c3813c7a89e8 P b98ed2c913804afe994cafecc1994347 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:59:46.199739 7943 raft_consensus.cc:729] T 3c679f4befa14b1d8b08c3813c7a89e8 P b98ed2c913804afe994cafecc1994347 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b98ed2c913804afe994cafecc1994347, State: Initialized, Role: FOLLOWER
I20260806 02:59:46.200006 7943 consensus_queue.cc:260] T 3c679f4befa14b1d8b08c3813c7a89e8 P b98ed2c913804afe994cafecc1994347 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98ed2c913804afe994cafecc1994347" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 35105 } } peers { permanent_uuid: "2fd6a06c3426465484a699dedffbf3b4" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 46395 } } peers { permanent_uuid: "67c6964d17f443dcacfb6a01a0945be4" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 38549 } }
I20260806 02:59:46.200637 7943 ts_tablet_manager.cc:1428] T 3c679f4befa14b1d8b08c3813c7a89e8 P b98ed2c913804afe994cafecc1994347: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260806 02:59:46.201562 7941 tablet_bootstrap.cc:492] T 3c679f4befa14b1d8b08c3813c7a89e8 P 2fd6a06c3426465484a699dedffbf3b4: Bootstrap starting.
I20260806 02:59:46.204437 7941 tablet_bootstrap.cc:654] T 3c679f4befa14b1d8b08c3813c7a89e8 P 2fd6a06c3426465484a699dedffbf3b4: Neither blocks nor log segments found. Creating new log.
I20260806 02:59:46.207242 7942 tablet_bootstrap.cc:492] T 3c679f4befa14b1d8b08c3813c7a89e8 P 67c6964d17f443dcacfb6a01a0945be4: Bootstrap starting.
I20260806 02:59:46.207283 7941 tablet_bootstrap.cc:492] T 3c679f4befa14b1d8b08c3813c7a89e8 P 2fd6a06c3426465484a699dedffbf3b4: No bootstrap required, opened a new log
I20260806 02:59:46.207485 7941 ts_tablet_manager.cc:1397] T 3c679f4befa14b1d8b08c3813c7a89e8 P 2fd6a06c3426465484a699dedffbf3b4: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260806 02:59:46.208272 7941 raft_consensus.cc:348] T 3c679f4befa14b1d8b08c3813c7a89e8 P 2fd6a06c3426465484a699dedffbf3b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98ed2c913804afe994cafecc1994347" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 35105 } } peers { permanent_uuid: "2fd6a06c3426465484a699dedffbf3b4" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 46395 } } peers { permanent_uuid: "67c6964d17f443dcacfb6a01a0945be4" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 38549 } }
I20260806 02:59:46.208590 7941 raft_consensus.cc:374] T 3c679f4befa14b1d8b08c3813c7a89e8 P 2fd6a06c3426465484a699dedffbf3b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:59:46.208832 7941 raft_consensus.cc:729] T 3c679f4befa14b1d8b08c3813c7a89e8 P 2fd6a06c3426465484a699dedffbf3b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fd6a06c3426465484a699dedffbf3b4, State: Initialized, Role: FOLLOWER
I20260806 02:59:46.209146 7941 consensus_queue.cc:260] T 3c679f4befa14b1d8b08c3813c7a89e8 P 2fd6a06c3426465484a699dedffbf3b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98ed2c913804afe994cafecc1994347" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 35105 } } peers { permanent_uuid: "2fd6a06c3426465484a699dedffbf3b4" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 46395 } } peers { permanent_uuid: "67c6964d17f443dcacfb6a01a0945be4" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 38549 } }
I20260806 02:59:46.209319 7942 tablet_bootstrap.cc:654] T 3c679f4befa14b1d8b08c3813c7a89e8 P 67c6964d17f443dcacfb6a01a0945be4: Neither blocks nor log segments found. Creating new log.
I20260806 02:59:46.209810 7941 ts_tablet_manager.cc:1428] T 3c679f4befa14b1d8b08c3813c7a89e8 P 2fd6a06c3426465484a699dedffbf3b4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260806 02:59:46.212061 7942 tablet_bootstrap.cc:492] T 3c679f4befa14b1d8b08c3813c7a89e8 P 67c6964d17f443dcacfb6a01a0945be4: No bootstrap required, opened a new log
I20260806 02:59:46.212268 7942 ts_tablet_manager.cc:1397] T 3c679f4befa14b1d8b08c3813c7a89e8 P 67c6964d17f443dcacfb6a01a0945be4: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260806 02:59:46.213173 7942 raft_consensus.cc:348] T 3c679f4befa14b1d8b08c3813c7a89e8 P 67c6964d17f443dcacfb6a01a0945be4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98ed2c913804afe994cafecc1994347" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 35105 } } peers { permanent_uuid: "2fd6a06c3426465484a699dedffbf3b4" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 46395 } } peers { permanent_uuid: "67c6964d17f443dcacfb6a01a0945be4" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 38549 } }
I20260806 02:59:46.213503 7942 raft_consensus.cc:374] T 3c679f4befa14b1d8b08c3813c7a89e8 P 67c6964d17f443dcacfb6a01a0945be4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:59:46.213603 7942 raft_consensus.cc:729] T 3c679f4befa14b1d8b08c3813c7a89e8 P 67c6964d17f443dcacfb6a01a0945be4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67c6964d17f443dcacfb6a01a0945be4, State: Initialized, Role: FOLLOWER
I20260806 02:59:46.213951 7942 consensus_queue.cc:260] T 3c679f4befa14b1d8b08c3813c7a89e8 P 67c6964d17f443dcacfb6a01a0945be4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98ed2c913804afe994cafecc1994347" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 35105 } } peers { permanent_uuid: "2fd6a06c3426465484a699dedffbf3b4" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 46395 } } peers { permanent_uuid: "67c6964d17f443dcacfb6a01a0945be4" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 38549 } }
I20260806 02:59:46.215265 7942 ts_tablet_manager.cc:1428] T 3c679f4befa14b1d8b08c3813c7a89e8 P 67c6964d17f443dcacfb6a01a0945be4: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260806 02:59:46.418215 7918 raft_consensus.cc:482] T 3c679f4befa14b1d8b08c3813c7a89e8 P b98ed2c913804afe994cafecc1994347 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:59:46.418514 7918 raft_consensus.cc:504] T 3c679f4befa14b1d8b08c3813c7a89e8 P b98ed2c913804afe994cafecc1994347 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98ed2c913804afe994cafecc1994347" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 35105 } } peers { permanent_uuid: "2fd6a06c3426465484a699dedffbf3b4" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 46395 } } peers { permanent_uuid: "67c6964d17f443dcacfb6a01a0945be4" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 38549 } }
I20260806 02:59:46.419483 7918 leader_election.cc:288] T 3c679f4befa14b1d8b08c3813c7a89e8 P b98ed2c913804afe994cafecc1994347 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2fd6a06c3426465484a699dedffbf3b4 (127.31.74.194:46395), 67c6964d17f443dcacfb6a01a0945be4 (127.31.74.195:38549)
I20260806 02:59:46.420156 7684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c679f4befa14b1d8b08c3813c7a89e8" candidate_uuid: "b98ed2c913804afe994cafecc1994347" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fd6a06c3426465484a699dedffbf3b4" is_pre_election: true
I20260806 02:59:46.420217 7828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c679f4befa14b1d8b08c3813c7a89e8" candidate_uuid: "b98ed2c913804afe994cafecc1994347" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67c6964d17f443dcacfb6a01a0945be4" is_pre_election: true
I20260806 02:59:46.420513 7828 raft_consensus.cc:2454] T 3c679f4befa14b1d8b08c3813c7a89e8 P 67c6964d17f443dcacfb6a01a0945be4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b98ed2c913804afe994cafecc1994347 in term 0.
I20260806 02:59:46.420531 7684 raft_consensus.cc:2454] T 3c679f4befa14b1d8b08c3813c7a89e8 P 2fd6a06c3426465484a699dedffbf3b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b98ed2c913804afe994cafecc1994347 in term 0.
I20260806 02:59:46.421047 7465 leader_election.cc:302] T 3c679f4befa14b1d8b08c3813c7a89e8 P b98ed2c913804afe994cafecc1994347 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67c6964d17f443dcacfb6a01a0945be4, b98ed2c913804afe994cafecc1994347; no voters:
I20260806 02:59:46.421373 7918 raft_consensus.cc:2781] T 3c679f4befa14b1d8b08c3813c7a89e8 P b98ed2c913804afe994cafecc1994347 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:59:46.421458 7918 raft_consensus.cc:482] T 3c679f4befa14b1d8b08c3813c7a89e8 P b98ed2c913804afe994cafecc1994347 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:59:46.421574 7918 raft_consensus.cc:3037] T 3c679f4befa14b1d8b08c3813c7a89e8 P b98ed2c913804afe994cafecc1994347 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:59:46.423774 7918 raft_consensus.cc:504] T 3c679f4befa14b1d8b08c3813c7a89e8 P b98ed2c913804afe994cafecc1994347 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98ed2c913804afe994cafecc1994347" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 35105 } } peers { permanent_uuid: "2fd6a06c3426465484a699dedffbf3b4" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 46395 } } peers { permanent_uuid: "67c6964d17f443dcacfb6a01a0945be4" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 38549 } }
I20260806 02:59:46.424455 7918 leader_election.cc:288] T 3c679f4befa14b1d8b08c3813c7a89e8 P b98ed2c913804afe994cafecc1994347 [CANDIDATE]: Term 1 election: Requested vote from peers 2fd6a06c3426465484a699dedffbf3b4 (127.31.74.194:46395), 67c6964d17f443dcacfb6a01a0945be4 (127.31.74.195:38549)
I20260806 02:59:46.424988 7828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c679f4befa14b1d8b08c3813c7a89e8" candidate_uuid: "b98ed2c913804afe994cafecc1994347" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67c6964d17f443dcacfb6a01a0945be4"
I20260806 02:59:46.425040 7684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c679f4befa14b1d8b08c3813c7a89e8" candidate_uuid: "b98ed2c913804afe994cafecc1994347" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fd6a06c3426465484a699dedffbf3b4"
I20260806 02:59:46.425187 7828 raft_consensus.cc:3037] T 3c679f4befa14b1d8b08c3813c7a89e8 P 67c6964d17f443dcacfb6a01a0945be4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:59:46.425293 7684 raft_consensus.cc:3037] T 3c679f4befa14b1d8b08c3813c7a89e8 P 2fd6a06c3426465484a699dedffbf3b4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:59:46.427374 7828 raft_consensus.cc:2454] T 3c679f4befa14b1d8b08c3813c7a89e8 P 67c6964d17f443dcacfb6a01a0945be4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b98ed2c913804afe994cafecc1994347 in term 1.
I20260806 02:59:46.427776 7684 raft_consensus.cc:2454] T 3c679f4befa14b1d8b08c3813c7a89e8 P 2fd6a06c3426465484a699dedffbf3b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b98ed2c913804afe994cafecc1994347 in term 1.
I20260806 02:59:46.427856 7465 leader_election.cc:302] T 3c679f4befa14b1d8b08c3813c7a89e8 P b98ed2c913804afe994cafecc1994347 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67c6964d17f443dcacfb6a01a0945be4, b98ed2c913804afe994cafecc1994347; no voters:
I20260806 02:59:46.428126 7918 raft_consensus.cc:2781] T 3c679f4befa14b1d8b08c3813c7a89e8 P b98ed2c913804afe994cafecc1994347 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:59:46.428328 7918 raft_consensus.cc:686] T 3c679f4befa14b1d8b08c3813c7a89e8 P b98ed2c913804afe994cafecc1994347 [term 1 LEADER]: Becoming Leader. State: Replica: b98ed2c913804afe994cafecc1994347, State: Running, Role: LEADER
I20260806 02:59:46.428582 7918 consensus_queue.cc:237] T 3c679f4befa14b1d8b08c3813c7a89e8 P b98ed2c913804afe994cafecc1994347 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98ed2c913804afe994cafecc1994347" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 35105 } } peers { permanent_uuid: "2fd6a06c3426465484a699dedffbf3b4" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 46395 } } peers { permanent_uuid: "67c6964d17f443dcacfb6a01a0945be4" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 38549 } }
I20260806 02:59:46.432000 7376 catalog_manager.cc:5495] T 3c679f4befa14b1d8b08c3813c7a89e8 P b98ed2c913804afe994cafecc1994347 reported cstate change: term changed from 0 to 1, leader changed from <none> to b98ed2c913804afe994cafecc1994347 (127.31.74.193). New cstate: current_term: 1 leader_uuid: "b98ed2c913804afe994cafecc1994347" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98ed2c913804afe994cafecc1994347" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 35105 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2fd6a06c3426465484a699dedffbf3b4" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 46395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67c6964d17f443dcacfb6a01a0945be4" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 38549 } health_report { overall_health: UNKNOWN } } }
I20260806 02:59:46.457257 7897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:46.451356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46046 (local address 127.31.74.193:35105)
0806 02:59:46.451624 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:46.451631 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:46.452090 (+ 459us) server_negotiation.cc:407] Connection header received
0806 02:59:46.452348 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:46.452352 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:46.452423 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:46.452529 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:59:46.452930 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:46.453774 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:46.455338 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:46.455771 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:46.455924 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:46.456375 (+ 451us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:59:46.456493 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:59:46.456593 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:46.456752 (+ 159us) server_negotiation.cc:299] Negotiation successful
0806 02:59:46.456837 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260806 02:59:46.460513 7684 raft_consensus.cc:1261] T 3c679f4befa14b1d8b08c3813c7a89e8 P 2fd6a06c3426465484a699dedffbf3b4 [term 1 FOLLOWER]: Refusing update from remote peer b98ed2c913804afe994cafecc1994347: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:59:46.460513 7828 raft_consensus.cc:1261] T 3c679f4befa14b1d8b08c3813c7a89e8 P 67c6964d17f443dcacfb6a01a0945be4 [term 1 FOLLOWER]: Refusing update from remote peer b98ed2c913804afe994cafecc1994347: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:59:46.461272 7918 consensus_queue.cc:1035] T 3c679f4befa14b1d8b08c3813c7a89e8 P b98ed2c913804afe994cafecc1994347 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fd6a06c3426465484a699dedffbf3b4" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 46395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:59:46.461594 7922 consensus_queue.cc:1035] T 3c679f4befa14b1d8b08c3813c7a89e8 P b98ed2c913804afe994cafecc1994347 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67c6964d17f443dcacfb6a01a0945be4" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 38549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:59:46.475749 7375 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:59:46.475984 7375 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:59:46.479307 7375 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8643cb88550544428f8c7eee4349597e: Sending DeleteTablet for 3 replicas of tablet 3c679f4befa14b1d8b08c3813c7a89e8
I20260806 02:59:46.480230 7516 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c679f4befa14b1d8b08c3813c7a89e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:59:46 UTC) from {username='oryx', principal='oryx/127.31.74.254@KRBTEST.COM'} at 127.0.0.1:46018
I20260806 02:59:46.480297 7662 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c679f4befa14b1d8b08c3813c7a89e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:59:46 UTC) from {username='oryx', principal='oryx/127.31.74.254@KRBTEST.COM'} at 127.0.0.1:53200
I20260806 02:59:46.480574 7968 tablet_replica.cc:330] T 3c679f4befa14b1d8b08c3813c7a89e8 P b98ed2c913804afe994cafecc1994347: stopping tablet replica
I20260806 02:59:46.480616 7966 tablet_replica.cc:330] T 3c679f4befa14b1d8b08c3813c7a89e8 P 2fd6a06c3426465484a699dedffbf3b4: stopping tablet replica
I20260806 02:59:46.480846 7968 raft_consensus.cc:2229] T 3c679f4befa14b1d8b08c3813c7a89e8 P b98ed2c913804afe994cafecc1994347 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:59:46.480851 7966 raft_consensus.cc:2229] T 3c679f4befa14b1d8b08c3813c7a89e8 P 2fd6a06c3426465484a699dedffbf3b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:59:46.481132 7966 raft_consensus.cc:2258] T 3c679f4befa14b1d8b08c3813c7a89e8 P 2fd6a06c3426465484a699dedffbf3b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:59:46.481230 7968 raft_consensus.cc:2258] T 3c679f4befa14b1d8b08c3813c7a89e8 P b98ed2c913804afe994cafecc1994347 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:59:46.481913 7803 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c679f4befa14b1d8b08c3813c7a89e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:59:46 UTC) from {username='oryx', principal='oryx/127.31.74.254@KRBTEST.COM'} at 127.0.0.1:59362
I20260806 02:59:46.481913 7375 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44480:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260806 02:59:46.482213 7967 tablet_replica.cc:330] T 3c679f4befa14b1d8b08c3813c7a89e8 P 67c6964d17f443dcacfb6a01a0945be4: stopping tablet replica
I20260806 02:59:46.482362 7968 ts_tablet_manager.cc:1905] T 3c679f4befa14b1d8b08c3813c7a89e8 P b98ed2c913804afe994cafecc1994347: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:59:46.482414 7967 raft_consensus.cc:2229] T 3c679f4befa14b1d8b08c3813c7a89e8 P 67c6964d17f443dcacfb6a01a0945be4 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260806 02:59:46.482640 7375 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:59:46.482698 7967 raft_consensus.cc:2258] T 3c679f4befa14b1d8b08c3813c7a89e8 P 67c6964d17f443dcacfb6a01a0945be4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:59:46.483768 7967 ts_tablet_manager.cc:1905] T 3c679f4befa14b1d8b08c3813c7a89e8 P 67c6964d17f443dcacfb6a01a0945be4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:59:46.484637 7966 ts_tablet_manager.cc:1905] T 3c679f4befa14b1d8b08c3813c7a89e8 P 2fd6a06c3426465484a699dedffbf3b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:59:46.489840 7968 ts_tablet_manager.cc:1918] T 3c679f4befa14b1d8b08c3813c7a89e8 P b98ed2c913804afe994cafecc1994347: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:59:46.490015 7968 log.cc:1196] T 3c679f4befa14b1d8b08c3813c7a89e8 P b98ed2c913804afe994cafecc1994347: Deleting WAL directory at /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/minicluster-data/ts-0/wal/wals/3c679f4befa14b1d8b08c3813c7a89e8
I20260806 02:59:46.490849 7968 ts_tablet_manager.cc:1939] T 3c679f4befa14b1d8b08c3813c7a89e8 P b98ed2c913804afe994cafecc1994347: Deleting consensus metadata
I20260806 02:59:46.491041 7662 tablet_service.cc:1467] Processing CreateTablet for tablet f1b99c255cb648dfaf8d6275ff014a0d (DEFAULT_TABLE table=test-table [id=4694d022e2d4400aa5f9dc54ce206768]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:59:46.491148 7516 tablet_service.cc:1467] Processing CreateTablet for tablet f1b99c255cb648dfaf8d6275ff014a0d (DEFAULT_TABLE table=test-table [id=4694d022e2d4400aa5f9dc54ce206768]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:59:46.491575 7516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1b99c255cb648dfaf8d6275ff014a0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:59:46.491575 7662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1b99c255cb648dfaf8d6275ff014a0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:59:46.492111 7362 catalog_manager.cc:4841] TS b98ed2c913804afe994cafecc1994347 (127.31.74.193:35105): tablet 3c679f4befa14b1d8b08c3813c7a89e8 (table test-table [id=d834d902cce54ca495abb2918dd25429]) successfully deleted
I20260806 02:59:46.495749 7803 tablet_service.cc:1467] Processing CreateTablet for tablet f1b99c255cb648dfaf8d6275ff014a0d (DEFAULT_TABLE table=test-table [id=4694d022e2d4400aa5f9dc54ce206768]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:59:46.496191 7803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1b99c255cb648dfaf8d6275ff014a0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:59:46.496349 7943 tablet_bootstrap.cc:492] T f1b99c255cb648dfaf8d6275ff014a0d P b98ed2c913804afe994cafecc1994347: Bootstrap starting.
I20260806 02:59:46.497752 7943 tablet_bootstrap.cc:654] T f1b99c255cb648dfaf8d6275ff014a0d P b98ed2c913804afe994cafecc1994347: Neither blocks nor log segments found. Creating new log.
I20260806 02:59:46.498247 7967 ts_tablet_manager.cc:1918] T 3c679f4befa14b1d8b08c3813c7a89e8 P 67c6964d17f443dcacfb6a01a0945be4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:59:46.498431 7967 log.cc:1196] T 3c679f4befa14b1d8b08c3813c7a89e8 P 67c6964d17f443dcacfb6a01a0945be4: Deleting WAL directory at /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/minicluster-data/ts-2/wal/wals/3c679f4befa14b1d8b08c3813c7a89e8
I20260806 02:59:46.491600 7966 ts_tablet_manager.cc:1918] T 3c679f4befa14b1d8b08c3813c7a89e8 P 2fd6a06c3426465484a699dedffbf3b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:59:46.498893 7966 log.cc:1196] T 3c679f4befa14b1d8b08c3813c7a89e8 P 2fd6a06c3426465484a699dedffbf3b4: Deleting WAL directory at /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/minicluster-data/ts-1/wal/wals/3c679f4befa14b1d8b08c3813c7a89e8
I20260806 02:59:46.498916 7967 ts_tablet_manager.cc:1939] T 3c679f4befa14b1d8b08c3813c7a89e8 P 67c6964d17f443dcacfb6a01a0945be4: Deleting consensus metadata
I20260806 02:59:46.498921 7941 tablet_bootstrap.cc:492] T f1b99c255cb648dfaf8d6275ff014a0d P 2fd6a06c3426465484a699dedffbf3b4: Bootstrap starting.
I20260806 02:59:46.499531 7966 ts_tablet_manager.cc:1939] T 3c679f4befa14b1d8b08c3813c7a89e8 P 2fd6a06c3426465484a699dedffbf3b4: Deleting consensus metadata
I20260806 02:59:46.501034 7941 tablet_bootstrap.cc:654] T f1b99c255cb648dfaf8d6275ff014a0d P 2fd6a06c3426465484a699dedffbf3b4: Neither blocks nor log segments found. Creating new log.
I20260806 02:59:46.501230 7363 catalog_manager.cc:4841] TS 2fd6a06c3426465484a699dedffbf3b4 (127.31.74.194:46395): tablet 3c679f4befa14b1d8b08c3813c7a89e8 (table test-table [id=d834d902cce54ca495abb2918dd25429]) successfully deleted
I20260806 02:59:46.502257 7942 tablet_bootstrap.cc:492] T f1b99c255cb648dfaf8d6275ff014a0d P 67c6964d17f443dcacfb6a01a0945be4: Bootstrap starting.
I20260806 02:59:46.502744 7360 catalog_manager.cc:4841] TS 67c6964d17f443dcacfb6a01a0945be4 (127.31.74.195:38549): tablet 3c679f4befa14b1d8b08c3813c7a89e8 (table test-table [id=d834d902cce54ca495abb2918dd25429]) successfully deleted
I20260806 02:59:46.503938 7943 tablet_bootstrap.cc:492] T f1b99c255cb648dfaf8d6275ff014a0d P b98ed2c913804afe994cafecc1994347: No bootstrap required, opened a new log
I20260806 02:59:46.504125 7943 ts_tablet_manager.cc:1397] T f1b99c255cb648dfaf8d6275ff014a0d P b98ed2c913804afe994cafecc1994347: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.002s
I20260806 02:59:46.504171 7942 tablet_bootstrap.cc:654] T f1b99c255cb648dfaf8d6275ff014a0d P 67c6964d17f443dcacfb6a01a0945be4: Neither blocks nor log segments found. Creating new log.
I20260806 02:59:46.504345 7941 tablet_bootstrap.cc:492] T f1b99c255cb648dfaf8d6275ff014a0d P 2fd6a06c3426465484a699dedffbf3b4: No bootstrap required, opened a new log
I20260806 02:59:46.504509 7941 ts_tablet_manager.cc:1397] T f1b99c255cb648dfaf8d6275ff014a0d P 2fd6a06c3426465484a699dedffbf3b4: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20260806 02:59:46.505115 7943 raft_consensus.cc:348] T f1b99c255cb648dfaf8d6275ff014a0d P b98ed2c913804afe994cafecc1994347 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98ed2c913804afe994cafecc1994347" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 35105 } } peers { permanent_uuid: "2fd6a06c3426465484a699dedffbf3b4" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 46395 } } peers { permanent_uuid: "67c6964d17f443dcacfb6a01a0945be4" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 38549 } }
I20260806 02:59:46.505406 7943 raft_consensus.cc:374] T f1b99c255cb648dfaf8d6275ff014a0d P b98ed2c913804afe994cafecc1994347 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:59:46.505347 7941 raft_consensus.cc:348] T f1b99c255cb648dfaf8d6275ff014a0d P 2fd6a06c3426465484a699dedffbf3b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98ed2c913804afe994cafecc1994347" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 35105 } } peers { permanent_uuid: "2fd6a06c3426465484a699dedffbf3b4" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 46395 } } peers { permanent_uuid: "67c6964d17f443dcacfb6a01a0945be4" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 38549 } }
I20260806 02:59:46.505546 7943 raft_consensus.cc:729] T f1b99c255cb648dfaf8d6275ff014a0d P b98ed2c913804afe994cafecc1994347 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b98ed2c913804afe994cafecc1994347, State: Initialized, Role: FOLLOWER
I20260806 02:59:46.505627 7941 raft_consensus.cc:374] T f1b99c255cb648dfaf8d6275ff014a0d P 2fd6a06c3426465484a699dedffbf3b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:59:46.505766 7941 raft_consensus.cc:729] T f1b99c255cb648dfaf8d6275ff014a0d P 2fd6a06c3426465484a699dedffbf3b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fd6a06c3426465484a699dedffbf3b4, State: Initialized, Role: FOLLOWER
I20260806 02:59:46.505890 7943 consensus_queue.cc:260] T f1b99c255cb648dfaf8d6275ff014a0d P b98ed2c913804afe994cafecc1994347 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98ed2c913804afe994cafecc1994347" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 35105 } } peers { permanent_uuid: "2fd6a06c3426465484a699dedffbf3b4" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 46395 } } peers { permanent_uuid: "67c6964d17f443dcacfb6a01a0945be4" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 38549 } }
I20260806 02:59:46.506038 7941 consensus_queue.cc:260] T f1b99c255cb648dfaf8d6275ff014a0d P 2fd6a06c3426465484a699dedffbf3b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98ed2c913804afe994cafecc1994347" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 35105 } } peers { permanent_uuid: "2fd6a06c3426465484a699dedffbf3b4" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 46395 } } peers { permanent_uuid: "67c6964d17f443dcacfb6a01a0945be4" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 38549 } }
I20260806 02:59:46.506395 7942 tablet_bootstrap.cc:492] T f1b99c255cb648dfaf8d6275ff014a0d P 67c6964d17f443dcacfb6a01a0945be4: No bootstrap required, opened a new log
I20260806 02:59:46.506546 7942 ts_tablet_manager.cc:1397] T f1b99c255cb648dfaf8d6275ff014a0d P 67c6964d17f443dcacfb6a01a0945be4: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260806 02:59:46.506556 7943 ts_tablet_manager.cc:1428] T f1b99c255cb648dfaf8d6275ff014a0d P b98ed2c913804afe994cafecc1994347: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260806 02:59:46.507404 7941 ts_tablet_manager.cc:1428] T f1b99c255cb648dfaf8d6275ff014a0d P 2fd6a06c3426465484a699dedffbf3b4: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260806 02:59:46.507441 7942 raft_consensus.cc:348] T f1b99c255cb648dfaf8d6275ff014a0d P 67c6964d17f443dcacfb6a01a0945be4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98ed2c913804afe994cafecc1994347" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 35105 } } peers { permanent_uuid: "2fd6a06c3426465484a699dedffbf3b4" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 46395 } } peers { permanent_uuid: "67c6964d17f443dcacfb6a01a0945be4" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 38549 } }
I20260806 02:59:46.507723 7942 raft_consensus.cc:374] T f1b99c255cb648dfaf8d6275ff014a0d P 67c6964d17f443dcacfb6a01a0945be4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:59:46.507840 7942 raft_consensus.cc:729] T f1b99c255cb648dfaf8d6275ff014a0d P 67c6964d17f443dcacfb6a01a0945be4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67c6964d17f443dcacfb6a01a0945be4, State: Initialized, Role: FOLLOWER
I20260806 02:59:46.508134 7942 consensus_queue.cc:260] T f1b99c255cb648dfaf8d6275ff014a0d P 67c6964d17f443dcacfb6a01a0945be4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98ed2c913804afe994cafecc1994347" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 35105 } } peers { permanent_uuid: "2fd6a06c3426465484a699dedffbf3b4" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 46395 } } peers { permanent_uuid: "67c6964d17f443dcacfb6a01a0945be4" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 38549 } }
I20260806 02:59:46.509423 7942 ts_tablet_manager.cc:1428] T f1b99c255cb648dfaf8d6275ff014a0d P 67c6964d17f443dcacfb6a01a0945be4: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260806 02:59:46.618860 7947 raft_consensus.cc:482] T f1b99c255cb648dfaf8d6275ff014a0d P 67c6964d17f443dcacfb6a01a0945be4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:59:46.619131 7947 raft_consensus.cc:504] T f1b99c255cb648dfaf8d6275ff014a0d P 67c6964d17f443dcacfb6a01a0945be4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98ed2c913804afe994cafecc1994347" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 35105 } } peers { permanent_uuid: "2fd6a06c3426465484a699dedffbf3b4" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 46395 } } peers { permanent_uuid: "67c6964d17f443dcacfb6a01a0945be4" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 38549 } }
I20260806 02:59:46.621726 7947 leader_election.cc:288] T f1b99c255cb648dfaf8d6275ff014a0d P 67c6964d17f443dcacfb6a01a0945be4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b98ed2c913804afe994cafecc1994347 (127.31.74.193:35105), 2fd6a06c3426465484a699dedffbf3b4 (127.31.74.194:46395)
I20260806 02:59:46.631328 7897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:46.622413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.195:41569 (local address 127.31.74.193:35105)
0806 02:59:46.622853 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:46.622861 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:46.623950 (+ 1089us) server_negotiation.cc:407] Connection header received
0806 02:59:46.624269 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:46.624277 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:46.624366 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:46.624436 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:59:46.625167 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:46.626144 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:46.628654 (+ 2510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:46.630104 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:46.630296 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:46.630403 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:46.630826 (+ 423us) server_negotiation.cc:299] Negotiation successful
0806 02:59:46.630996 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249}
I20260806 02:59:46.631639 7993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:46.622027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.194:46395 (local address 127.31.74.195:58129)
0806 02:59:46.624348 (+ 2321us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:46.624370 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:46.624575 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:46.625157 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:46.625162 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:46.625191 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:59:46.625558 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:46.625570 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:46.627136 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:46.627142 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:46.629092 (+ 1950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:46.629103 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:46.630640 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:46.630648 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:46.630871 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:46.630910 (+ 39us) client_negotiation.cc:769] Sending connection context
0806 02:59:46.631009 (+ 99us) client_negotiation.cc:240] Negotiation successful
0806 02:59:46.631155 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2182,"thread_start_us":1507,"threads_started":1}
I20260806 02:59:46.631975 7994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:46.622445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.195:58129 (local address 127.31.74.194:46395)
0806 02:59:46.623317 (+ 872us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:46.623324 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:46.624467 (+ 1143us) server_negotiation.cc:407] Connection header received
0806 02:59:46.624751 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:46.624757 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:46.624855 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:46.624921 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:59:46.625741 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:46.626976 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:46.629277 (+ 2301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:46.630348 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:46.630525 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:46.630637 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:46.631305 (+ 668us) server_negotiation.cc:299] Negotiation successful
0806 02:59:46.631551 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":728,"thread_start_us":558,"threads_started":1}
I20260806 02:59:46.632031 7992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:46.621975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:35105 (local address 127.31.74.195:41569)
0806 02:59:46.623829 (+ 1854us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:46.623854 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:46.624085 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:46.624665 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:46.624670 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:46.624716 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:59:46.624978 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:46.624989 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:46.626306 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:46.626316 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:46.628475 (+ 2159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:46.628491 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:46.630315 (+ 1824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:46.630322 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:46.630532 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:46.630574 (+ 42us) client_negotiation.cc:769] Sending connection context
0806 02:59:46.630828 (+ 254us) client_negotiation.cc:240] Negotiation successful
0806 02:59:46.631025 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1683,"thread_start_us":1995,"threads_started":1}
I20260806 02:59:46.632673 7684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1b99c255cb648dfaf8d6275ff014a0d" candidate_uuid: "67c6964d17f443dcacfb6a01a0945be4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fd6a06c3426465484a699dedffbf3b4" is_pre_election: true
I20260806 02:59:46.632961 7537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1b99c255cb648dfaf8d6275ff014a0d" candidate_uuid: "67c6964d17f443dcacfb6a01a0945be4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b98ed2c913804afe994cafecc1994347" is_pre_election: true
I20260806 02:59:46.633028 7684 raft_consensus.cc:2454] T f1b99c255cb648dfaf8d6275ff014a0d P 2fd6a06c3426465484a699dedffbf3b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67c6964d17f443dcacfb6a01a0945be4 in term 0.
I20260806 02:59:46.633309 7537 raft_consensus.cc:2454] T f1b99c255cb648dfaf8d6275ff014a0d P b98ed2c913804afe994cafecc1994347 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67c6964d17f443dcacfb6a01a0945be4 in term 0.
I20260806 02:59:46.633643 7753 leader_election.cc:302] T f1b99c255cb648dfaf8d6275ff014a0d P 67c6964d17f443dcacfb6a01a0945be4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fd6a06c3426465484a699dedffbf3b4, 67c6964d17f443dcacfb6a01a0945be4; no voters:
I20260806 02:59:46.634184 7947 raft_consensus.cc:2781] T f1b99c255cb648dfaf8d6275ff014a0d P 67c6964d17f443dcacfb6a01a0945be4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:59:46.634382 7947 raft_consensus.cc:482] T f1b99c255cb648dfaf8d6275ff014a0d P 67c6964d17f443dcacfb6a01a0945be4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:59:46.634501 7947 raft_consensus.cc:3037] T f1b99c255cb648dfaf8d6275ff014a0d P 67c6964d17f443dcacfb6a01a0945be4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:59:46.637989 7947 raft_consensus.cc:504] T f1b99c255cb648dfaf8d6275ff014a0d P 67c6964d17f443dcacfb6a01a0945be4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98ed2c913804afe994cafecc1994347" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 35105 } } peers { permanent_uuid: "2fd6a06c3426465484a699dedffbf3b4" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 46395 } } peers { permanent_uuid: "67c6964d17f443dcacfb6a01a0945be4" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 38549 } }
I20260806 02:59:46.638769 7947 leader_election.cc:288] T f1b99c255cb648dfaf8d6275ff014a0d P 67c6964d17f443dcacfb6a01a0945be4 [CANDIDATE]: Term 1 election: Requested vote from peers b98ed2c913804afe994cafecc1994347 (127.31.74.193:35105), 2fd6a06c3426465484a699dedffbf3b4 (127.31.74.194:46395)
I20260806 02:59:46.639362 7537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1b99c255cb648dfaf8d6275ff014a0d" candidate_uuid: "67c6964d17f443dcacfb6a01a0945be4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b98ed2c913804afe994cafecc1994347"
I20260806 02:59:46.639706 7537 raft_consensus.cc:3037] T f1b99c255cb648dfaf8d6275ff014a0d P b98ed2c913804afe994cafecc1994347 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:59:46.642925 7684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1b99c255cb648dfaf8d6275ff014a0d" candidate_uuid: "67c6964d17f443dcacfb6a01a0945be4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fd6a06c3426465484a699dedffbf3b4"
I20260806 02:59:46.643181 7684 raft_consensus.cc:3037] T f1b99c255cb648dfaf8d6275ff014a0d P 2fd6a06c3426465484a699dedffbf3b4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:59:46.643378 7537 raft_consensus.cc:2454] T f1b99c255cb648dfaf8d6275ff014a0d P b98ed2c913804afe994cafecc1994347 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67c6964d17f443dcacfb6a01a0945be4 in term 1.
I20260806 02:59:46.643946 7752 leader_election.cc:302] T f1b99c255cb648dfaf8d6275ff014a0d P 67c6964d17f443dcacfb6a01a0945be4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67c6964d17f443dcacfb6a01a0945be4, b98ed2c913804afe994cafecc1994347; no voters:
I20260806 02:59:46.644328 7947 raft_consensus.cc:2781] T f1b99c255cb648dfaf8d6275ff014a0d P 67c6964d17f443dcacfb6a01a0945be4 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:59:46.644882 7947 raft_consensus.cc:686] T f1b99c255cb648dfaf8d6275ff014a0d P 67c6964d17f443dcacfb6a01a0945be4 [term 1 LEADER]: Becoming Leader. State: Replica: 67c6964d17f443dcacfb6a01a0945be4, State: Running, Role: LEADER
I20260806 02:59:46.645331 7947 consensus_queue.cc:237] T f1b99c255cb648dfaf8d6275ff014a0d P 67c6964d17f443dcacfb6a01a0945be4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98ed2c913804afe994cafecc1994347" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 35105 } } peers { permanent_uuid: "2fd6a06c3426465484a699dedffbf3b4" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 46395 } } peers { permanent_uuid: "67c6964d17f443dcacfb6a01a0945be4" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 38549 } }
I20260806 02:59:46.646888 7684 raft_consensus.cc:2454] T f1b99c255cb648dfaf8d6275ff014a0d P 2fd6a06c3426465484a699dedffbf3b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67c6964d17f443dcacfb6a01a0945be4 in term 1.
I20260806 02:59:46.649672 7376 catalog_manager.cc:5495] T f1b99c255cb648dfaf8d6275ff014a0d P 67c6964d17f443dcacfb6a01a0945be4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 67c6964d17f443dcacfb6a01a0945be4 (127.31.74.195). New cstate: current_term: 1 leader_uuid: "67c6964d17f443dcacfb6a01a0945be4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98ed2c913804afe994cafecc1994347" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 35105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2fd6a06c3426465484a699dedffbf3b4" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 46395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67c6964d17f443dcacfb6a01a0945be4" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 38549 } health_report { overall_health: HEALTHY } } }
I20260806 02:59:46.697170 7997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:46.684290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59370 (local address 127.31.74.195:38549)
0806 02:59:46.689982 (+ 5692us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:46.689989 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:46.690015 (+ 26us) server_negotiation.cc:407] Connection header received
0806 02:59:46.690147 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:46.690156 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:46.690311 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:46.690506 (+ 195us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:59:46.691021 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:46.692372 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:46.693971 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:46.694779 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:46.694905 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:46.695562 (+ 657us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:59:46.696417 (+ 855us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:59:46.696572 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:46.696662 (+ 90us) server_negotiation.cc:299] Negotiation successful
0806 02:59:46.696752 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5540,"thread_start_us":5350,"threads_started":1}
I20260806 02:59:46.714838 7897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:46.709117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.195:39595 (local address 127.31.74.193:35105)
0806 02:59:46.709657 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:46.709666 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:46.710051 (+ 385us) server_negotiation.cc:407] Connection header received
0806 02:59:46.710287 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:46.710291 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:46.710365 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:46.710470 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:59:46.710985 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:46.711698 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:46.713081 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:46.713483 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:46.713650 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:46.714093 (+ 443us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:59:46.714191 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:59:46.714288 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:46.714429 (+ 141us) server_negotiation.cc:299] Negotiation successful
0806 02:59:46.714510 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147}
I20260806 02:59:46.714964 8000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:46.708868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:35105 (local address 127.31.74.195:39595)
0806 02:59:46.709939 (+ 1071us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:46.709955 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:46.710131 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:46.710488 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:46.710492 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:46.710504 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 02:59:46.710840 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:46.710848 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:46.711847 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:46.711852 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:46.712942 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:46.712953 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:46.713695 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:46.713704 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:46.713855 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:46.713910 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 02:59:46.714292 (+ 382us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 02:59:46.714305 (+ 13us) client_negotiation.cc:769] Sending connection context
0806 02:59:46.714405 (+ 100us) client_negotiation.cc:240] Negotiation successful
0806 02:59:46.714545 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":922,"thread_start_us":722,"threads_started":1}
I20260806 02:59:46.724742 7537 raft_consensus.cc:1261] T f1b99c255cb648dfaf8d6275ff014a0d P b98ed2c913804afe994cafecc1994347 [term 1 FOLLOWER]: Refusing update from remote peer 67c6964d17f443dcacfb6a01a0945be4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:59:46.725423 7685 raft_consensus.cc:1261] T f1b99c255cb648dfaf8d6275ff014a0d P 2fd6a06c3426465484a699dedffbf3b4 [term 1 FOLLOWER]: Refusing update from remote peer 67c6964d17f443dcacfb6a01a0945be4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:59:46.725634 7995 consensus_queue.cc:1035] T f1b99c255cb648dfaf8d6275ff014a0d P 67c6964d17f443dcacfb6a01a0945be4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b98ed2c913804afe994cafecc1994347" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:59:46.726146 7947 consensus_queue.cc:1035] T f1b99c255cb648dfaf8d6275ff014a0d P 67c6964d17f443dcacfb6a01a0945be4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fd6a06c3426465484a699dedffbf3b4" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 46395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:59:46.759452 7997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:46.753817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.193:48245 (local address 127.31.74.195:38549)
0806 02:59:46.754032 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:46.754039 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:46.754147 (+ 108us) server_negotiation.cc:407] Connection header received
0806 02:59:46.754408 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:46.754414 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:46.754497 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:46.754637 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:59:46.755099 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:46.755928 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:46.757458 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:46.758030 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:46.758178 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:46.758631 (+ 453us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:59:46.758791 (+ 160us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:59:46.758889 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:46.759045 (+ 156us) server_negotiation.cc:299] Negotiation successful
0806 02:59:46.759133 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 02:59:46.759563 8006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:46.753367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.195:38549 (local address 127.31.74.193:48245)
0806 02:59:46.754036 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:46.754064 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:46.754274 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:46.754654 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:46.754659 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:46.754671 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 02:59:46.754950 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:46.754960 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:46.756096 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:46.756102 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:46.757290 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:46.757303 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:46.758202 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:46.758209 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:46.758385 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:46.758426 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 02:59:46.758892 (+ 466us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 02:59:46.758907 (+ 15us) client_negotiation.cc:769] Sending connection context
0806 02:59:46.759022 (+ 115us) client_negotiation.cc:240] Negotiation successful
0806 02:59:46.759187 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":330,"threads_started":1}
I20260806 02:59:46.798108 7374 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:59:46.798357 7374 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:59:46.802115 7374 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8643cb88550544428f8c7eee4349597e: Sending DeleteTablet for 3 replicas of tablet f1b99c255cb648dfaf8d6275ff014a0d
I20260806 02:59:46.803157 7516 tablet_service.cc:1514] Processing DeleteTablet for tablet f1b99c255cb648dfaf8d6275ff014a0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:59:46 UTC) from {username='oryx', principal='oryx/127.31.74.254@KRBTEST.COM'} at 127.0.0.1:46018
I20260806 02:59:46.803174 7662 tablet_service.cc:1514] Processing DeleteTablet for tablet f1b99c255cb648dfaf8d6275ff014a0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:59:46 UTC) from {username='oryx', principal='oryx/127.31.74.254@KRBTEST.COM'} at 127.0.0.1:53200
I20260806 02:59:46.803177 7803 tablet_service.cc:1514] Processing DeleteTablet for tablet f1b99c255cb648dfaf8d6275ff014a0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:59:46 UTC) from {username='oryx', principal='oryx/127.31.74.254@KRBTEST.COM'} at 127.0.0.1:59362
I20260806 02:59:46.803519 7966 tablet_replica.cc:330] T f1b99c255cb648dfaf8d6275ff014a0d P 2fd6a06c3426465484a699dedffbf3b4: stopping tablet replica
I20260806 02:59:46.803529 7967 tablet_replica.cc:330] T f1b99c255cb648dfaf8d6275ff014a0d P 67c6964d17f443dcacfb6a01a0945be4: stopping tablet replica
I20260806 02:59:46.803540 7968 tablet_replica.cc:330] T f1b99c255cb648dfaf8d6275ff014a0d P b98ed2c913804afe994cafecc1994347: stopping tablet replica
I20260806 02:59:46.803718 7966 raft_consensus.cc:2229] T f1b99c255cb648dfaf8d6275ff014a0d P 2fd6a06c3426465484a699dedffbf3b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:59:46.803726 7967 raft_consensus.cc:2229] T f1b99c255cb648dfaf8d6275ff014a0d P 67c6964d17f443dcacfb6a01a0945be4 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:59:46.803782 7968 raft_consensus.cc:2229] T f1b99c255cb648dfaf8d6275ff014a0d P b98ed2c913804afe994cafecc1994347 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:59:46.803952 7966 raft_consensus.cc:2258] T f1b99c255cb648dfaf8d6275ff014a0d P 2fd6a06c3426465484a699dedffbf3b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:59:46.804060 7968 raft_consensus.cc:2258] T f1b99c255cb648dfaf8d6275ff014a0d P b98ed2c913804afe994cafecc1994347 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:59:46.804312 7967 raft_consensus.cc:2258] T f1b99c255cb648dfaf8d6275ff014a0d P 67c6964d17f443dcacfb6a01a0945be4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:59:46.805888 7968 ts_tablet_manager.cc:1905] T f1b99c255cb648dfaf8d6275ff014a0d P b98ed2c913804afe994cafecc1994347: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:59:46.806706 7966 ts_tablet_manager.cc:1905] T f1b99c255cb648dfaf8d6275ff014a0d P 2fd6a06c3426465484a699dedffbf3b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:59:46.807143 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 7449
I20260806 02:59:46.811517 7967 ts_tablet_manager.cc:1905] T f1b99c255cb648dfaf8d6275ff014a0d P 67c6964d17f443dcacfb6a01a0945be4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:59:46.816403 7966 ts_tablet_manager.cc:1918] T f1b99c255cb648dfaf8d6275ff014a0d P 2fd6a06c3426465484a699dedffbf3b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:59:46.816597 7966 log.cc:1196] T f1b99c255cb648dfaf8d6275ff014a0d P 2fd6a06c3426465484a699dedffbf3b4: Deleting WAL directory at /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/minicluster-data/ts-1/wal/wals/f1b99c255cb648dfaf8d6275ff014a0d
I20260806 02:59:46.817248 7966 ts_tablet_manager.cc:1939] T f1b99c255cb648dfaf8d6275ff014a0d P 2fd6a06c3426465484a699dedffbf3b4: Deleting consensus metadata
I20260806 02:59:46.818982 7363 catalog_manager.cc:4841] TS 2fd6a06c3426465484a699dedffbf3b4 (127.31.74.194:46395): tablet f1b99c255cb648dfaf8d6275ff014a0d (table test-table [id=4694d022e2d4400aa5f9dc54ce206768]) successfully deleted
I20260806 02:59:46.821350 7967 ts_tablet_manager.cc:1918] T f1b99c255cb648dfaf8d6275ff014a0d P 67c6964d17f443dcacfb6a01a0945be4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:59:46.821532 7967 log.cc:1196] T f1b99c255cb648dfaf8d6275ff014a0d P 67c6964d17f443dcacfb6a01a0945be4: Deleting WAL directory at /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/minicluster-data/ts-2/wal/wals/f1b99c255cb648dfaf8d6275ff014a0d
I20260806 02:59:46.822145 7967 ts_tablet_manager.cc:1939] T f1b99c255cb648dfaf8d6275ff014a0d P 67c6964d17f443dcacfb6a01a0945be4: Deleting consensus metadata
I20260806 02:59:46.822273 7968 ts_tablet_manager.cc:1918] T f1b99c255cb648dfaf8d6275ff014a0d P b98ed2c913804afe994cafecc1994347: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:59:46.822482 7968 log.cc:1196] T f1b99c255cb648dfaf8d6275ff014a0d P b98ed2c913804afe994cafecc1994347: Deleting WAL directory at /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785985105255475-32043-0/minicluster-data/ts-0/wal/wals/f1b99c255cb648dfaf8d6275ff014a0d
I20260806 02:59:46.823031 7968 ts_tablet_manager.cc:1939] T f1b99c255cb648dfaf8d6275ff014a0d P b98ed2c913804afe994cafecc1994347: Deleting consensus metadata
I20260806 02:59:46.823815 7360 catalog_manager.cc:4841] TS 67c6964d17f443dcacfb6a01a0945be4 (127.31.74.195:38549): tablet f1b99c255cb648dfaf8d6275ff014a0d (table test-table [id=4694d022e2d4400aa5f9dc54ce206768]) successfully deleted
I20260806 02:59:46.824327 7362 catalog_manager.cc:4841] TS b98ed2c913804afe994cafecc1994347 (127.31.74.193:35105): tablet f1b99c255cb648dfaf8d6275ff014a0d (table test-table [id=4694d022e2d4400aa5f9dc54ce206768]) successfully deleted
Aug 06 02:59:46 dist-test-slave-dhph krb5kdc[7331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.74.193@KRBTEST.COM
I20260806 02:59:46.839708 7897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:46.808816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46060 (local address 127.31.74.193:35105)
0806 02:59:46.809019 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:46.809026 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:46.820157 (+ 11131us) server_negotiation.cc:407] Connection header received
0806 02:59:46.820409 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:46.820416 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:46.820507 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:46.820582 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:46.822770 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:46.823974 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:46.825837 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:46.826420 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:46.826590 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:46.831468 (+ 4878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:46.831487 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:46.831505 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:46.831550 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:46.835734 (+ 4184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:46.836650 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:46.836658 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:46.836661 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:46.836843 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:46.837310 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:46.837314 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:46.837317 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:46.837746 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:46.838361 (+ 615us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:46.838897 (+ 536us) server_negotiation.cc:299] Negotiation successful
0806 02:59:46.839249 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 02:59:46.840885 7583 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:59:47.006938 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 7449
I20260806 02:59:47.052831 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 7592
Aug 06 02:59:47 dist-test-slave-dhph krb5kdc[7331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.74.194@KRBTEST.COM
I20260806 02:59:47.072418 7994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:47.053720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53214 (local address 127.31.74.194:46395)
0806 02:59:47.053908 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:47.053917 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:47.054007 (+ 90us) server_negotiation.cc:407] Connection header received
0806 02:59:47.054235 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:47.054242 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:47.054323 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:47.054442 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:47.055876 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:47.056649 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:47.058020 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:47.058656 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:47.058852 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:47.064172 (+ 5320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:47.064193 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:47.064214 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:47.064265 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:47.068821 (+ 4556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:47.069679 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:47.069691 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:47.069695 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:47.069887 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:47.070341 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:47.070346 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:47.070351 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:47.071035 (+ 684us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:47.071400 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:47.071694 (+ 294us) server_negotiation.cc:299] Negotiation successful
0806 02:59:47.071978 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 02:59:47.073328 7725 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:59:47.177089 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 7592
I20260806 02:59:47.199322 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 7734
Aug 06 02:59:47 dist-test-slave-dhph krb5kdc[7331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.74.195@KRBTEST.COM
I20260806 02:59:47.218459 7997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:47.200292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59378 (local address 127.31.74.195:38549)
0806 02:59:47.200730 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:47.200735 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:47.200755 (+ 20us) server_negotiation.cc:407] Connection header received
0806 02:59:47.200845 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:47.200849 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:47.200911 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:47.201044 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:47.202557 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:47.203461 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:47.205023 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:47.205521 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:47.205666 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:47.210927 (+ 5261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:47.210943 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:47.210970 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:47.211017 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:47.214439 (+ 3422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:47.215548 (+ 1109us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:47.215563 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:47.215567 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:47.215772 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:47.216349 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:47.216354 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:47.216356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:47.216798 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:47.217157 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:47.217882 (+ 725us) server_negotiation.cc:299] Negotiation successful
0806 02:59:47.218115 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290}
I20260806 02:59:47.219225 7868 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:59:47.360767 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 7734
I20260806 02:59:47.387708 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 7344
I20260806 02:59:47.389046 7416 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:59:47.510349 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 7344
2026-08-06T02:59:47Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (8003 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785985105255475-32043-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:59:47 dist-test-slave-dhph krb5kdc[8017](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:59:47 dist-test-slave-dhph krb5kdc[8017](info): set up 2 sockets
Aug 06 02:59:47 dist-test-slave-dhph krb5kdc[8017](info): commencing operation
krb5kdc: starting...
W20260806 02:59:48.719460 32043 mini_kdc.cc:121] Time spent starting KDC: real 1.157s user 0.000s sys 0.043s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785985105255475-32043-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785985105255475-32043-0/krb5kdc/test-user.keytab.
Aug 06 02:59:48 dist-test-slave-dhph krb5kdc[8017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:59:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:59:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.74.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab.
Entry for principal kudu/127.31.74.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab.
I20260806 02:59:49.047672 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785985105255475-32043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785985105255475-32043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785985105255475-32043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:44831
--webserver_interface=127.31.74.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab
--principal=kudu/127.31.74.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:46775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.74.254:44831
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:59:49.418226 8030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:59:49.418565 8030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:59:49.418790 8030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:59:49.430295 8030 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:59:49.430445 8030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:59:49.430550 8030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:59:49.430708 8030 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:59:49.430794 8030 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:59:49.442231 8030 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:46775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785985105255475-32043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785985105255475-32043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.74.254:44831
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:44831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.74.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785985105255475-32043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:59:49.444034 8030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:59:49.445477 8030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:59:49.459061 8035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:49.459125 8036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:49.459172 8038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:59:49.460104 8030 server_base.cc:1034] running on GCE node
I20260806 02:59:49.461086 8030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:59:49.462774 8030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:59:49.464013 8030 hybrid_clock.cc:648] HybridClock initialized: now 1785985189463945 us; error 63 us; skew 500 ppm
I20260806 02:59:49.464453 8030 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-06T02:59:49Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (2029 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:59:49 dist-test-slave-dhph krb5kdc[8044](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:59:49 dist-test-slave-dhph krb5kdc[8044](info): set up 2 sockets
Aug 06 02:59:49 dist-test-slave-dhph krb5kdc[8044](info): commencing operation
krb5kdc: starting...
W20260806 02:59:50.827121 32043 mini_kdc.cc:121] Time spent starting KDC: real 1.235s user 0.003s sys 0.043s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/krb5kdc/test-user.keytab.
Aug 06 02:59:51 dist-test-slave-dhph krb5kdc[8044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:59:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:59:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.74.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab.
Entry for principal kudu/127.31.74.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab.
I20260806 02:59:51.156467 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:45993
--webserver_interface=127.31.74.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab
--principal=kudu/127.31.74.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:45635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.74.254:45993
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:59:51.530483 8058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:59:51.530947 8058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:59:51.531117 8058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:59:51.540194 8058 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:59:51.540380 8058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:59:51.540503 8058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:59:51.540603 8058 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:59:51.540689 8058 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:59:51.550954 8058 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:45635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.74.254:45993
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab
--principal=kudu/127.31.74.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:45993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.74.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:59:51.552671 8058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:59:51.554482 8058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:59:51.568578 8063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:51.568536 8066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:51.568626 8064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:59:51.569377 8058 server_base.cc:1034] running on GCE node
I20260806 02:59:51.570331 8058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:59:51.572248 8058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:59:51.573534 8058 hybrid_clock.cc:648] HybridClock initialized: now 1785985191573476 us; error 67 us; skew 500 ppm
I20260806 02:59:51.573995 8058 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:59:51 dist-test-slave-dhph krb5kdc[8044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985191, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:59:51.577836 8058 init.cc:377] Logged in from keytab as kudu/127.31.74.254@KRBTEST.COM (short username kudu)
I20260806 02:59:51.579613 8058 webserver.cc:460] Webserver started at http://127.31.74.254:41489/ using document root <none> and password file <none>
I20260806 02:59:51.580377 8058 fs_manager.cc:359] Metadata directory not provided
I20260806 02:59:51.580526 8058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:59:51.580857 8058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:59:51.584389 8058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/master-0/data/instance:
uuid: "9fa46e0fbb384d4c92aebae63e956734"
format_stamp: "Formatted at 2026-08-06 02:59:51 on dist-test-slave-dhph"
server_key: "67c197232d5de389895e3210224a5081"
server_key_iv: "03cdf1699ba40659ae59dbfda4d558fe"
server_key_version: "encryptionkey@0"
I20260806 02:59:51.585268 8058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/master-0/wal/instance:
uuid: "9fa46e0fbb384d4c92aebae63e956734"
format_stamp: "Formatted at 2026-08-06 02:59:51 on dist-test-slave-dhph"
server_key: "67c197232d5de389895e3210224a5081"
server_key_iv: "03cdf1699ba40659ae59dbfda4d558fe"
server_key_version: "encryptionkey@0"
I20260806 02:59:51.591840 8058 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260806 02:59:51.596048 8073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:51.597653 8058 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 02:59:51.597888 8058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/master-0/wal
uuid: "9fa46e0fbb384d4c92aebae63e956734"
format_stamp: "Formatted at 2026-08-06 02:59:51 on dist-test-slave-dhph"
server_key: "67c197232d5de389895e3210224a5081"
server_key_iv: "03cdf1699ba40659ae59dbfda4d558fe"
server_key_version: "encryptionkey@0"
I20260806 02:59:51.598131 8058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:59:51.633664 8058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:59:51.638475 8058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:59:51.638878 8058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:59:51.663235 8058 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.254:45993
I20260806 02:59:51.663273 8124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.254:45993 every 8 connection(s)
I20260806 02:59:51.664860 8058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
I20260806 02:59:51.668900 8125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:59:51.675067 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 8058
I20260806 02:59:51.675318 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/master-0/wal/instance
I20260806 02:59:51.676168 32043 external_mini_cluster.cc:1428] Setting key 4debbd090777c9a3a374183a08607aab
I20260806 02:59:51.681958 8125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fa46e0fbb384d4c92aebae63e956734: Bootstrap starting.
I20260806 02:59:51.686002 8125 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9fa46e0fbb384d4c92aebae63e956734: Neither blocks nor log segments found. Creating new log.
I20260806 02:59:51.687393 8125 log.cc:824] T 00000000000000000000000000000000 P 9fa46e0fbb384d4c92aebae63e956734: Log is configured to *not* fsync() on all Append() calls
I20260806 02:59:51.691128 8125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fa46e0fbb384d4c92aebae63e956734: No bootstrap required, opened a new log
Aug 06 02:59:51 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 02:59:51.701289 8125 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9fa46e0fbb384d4c92aebae63e956734 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fa46e0fbb384d4c92aebae63e956734" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 45993 } }
I20260806 02:59:51.701764 8125 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9fa46e0fbb384d4c92aebae63e956734 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:59:51.701900 8125 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9fa46e0fbb384d4c92aebae63e956734 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fa46e0fbb384d4c92aebae63e956734, State: Initialized, Role: FOLLOWER
I20260806 02:59:51.702883 8125 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9fa46e0fbb384d4c92aebae63e956734 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fa46e0fbb384d4c92aebae63e956734" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 45993 } }
I20260806 02:59:51.703190 8125 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9fa46e0fbb384d4c92aebae63e956734 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:59:51.703368 8125 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9fa46e0fbb384d4c92aebae63e956734 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:59:51.703601 8125 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9fa46e0fbb384d4c92aebae63e956734 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:59:51.707659 8128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:51.678794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33040 (local address 127.31.74.254:45993)
0806 02:59:51.679800 (+ 1006us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:51.679814 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:51.679860 (+ 46us) server_negotiation.cc:407] Connection header received
0806 02:59:51.681795 (+ 1935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:51.681820 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:51.682314 (+ 494us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:51.682762 (+ 448us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:51.684774 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:51.686499 (+ 1725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:51.688447 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:51.689892 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:51.690039 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:51.695585 (+ 5546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:51.695595 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:51.695612 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:51.695658 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:51.701277 (+ 5619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:51.702487 (+ 1210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:51.702498 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:51.702507 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:51.702814 (+ 307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:51.703268 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:51.703275 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:51.703288 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:51.704241 (+ 953us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:51.704623 (+ 382us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:51.705413 (+ 790us) server_negotiation.cc:299] Negotiation successful
0806 02:59:51.705971 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":759,"thread_start_us":487,"threads_started":1}
I20260806 02:59:51.707921 8125 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9fa46e0fbb384d4c92aebae63e956734 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fa46e0fbb384d4c92aebae63e956734" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 45993 } }
I20260806 02:59:51.708632 8125 leader_election.cc:302] T 00000000000000000000000000000000 P 9fa46e0fbb384d4c92aebae63e956734 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9fa46e0fbb384d4c92aebae63e956734; no voters:
I20260806 02:59:51.709707 8125 leader_election.cc:288] T 00000000000000000000000000000000 P 9fa46e0fbb384d4c92aebae63e956734 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:59:51.709903 8130 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9fa46e0fbb384d4c92aebae63e956734 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:59:51.712534 8125 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9fa46e0fbb384d4c92aebae63e956734 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:59:51.715485 8130 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9fa46e0fbb384d4c92aebae63e956734 [term 1 LEADER]: Becoming Leader. State: Replica: 9fa46e0fbb384d4c92aebae63e956734, State: Running, Role: LEADER
I20260806 02:59:51.716078 8130 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9fa46e0fbb384d4c92aebae63e956734 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fa46e0fbb384d4c92aebae63e956734" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 45993 } }
I20260806 02:59:51.719165 8132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fa46e0fbb384d4c92aebae63e956734 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9fa46e0fbb384d4c92aebae63e956734" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fa46e0fbb384d4c92aebae63e956734" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 45993 } } }
I20260806 02:59:51.719565 8133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fa46e0fbb384d4c92aebae63e956734 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9fa46e0fbb384d4c92aebae63e956734. Latest consensus state: current_term: 1 leader_uuid: "9fa46e0fbb384d4c92aebae63e956734" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fa46e0fbb384d4c92aebae63e956734" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 45993 } } }
I20260806 02:59:51.720613 8132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fa46e0fbb384d4c92aebae63e956734 [sys.catalog]: This master's current role is: LEADER
I20260806 02:59:51.720613 8133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fa46e0fbb384d4c92aebae63e956734 [sys.catalog]: This master's current role is: LEADER
I20260806 02:59:51.722133 8140 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:59:51.729748 8140 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:59:51.743239 8140 catalog_manager.cc:1347] Generated new cluster ID: 47947b18a85e4961b5404e2791103b41
I20260806 02:59:51.743374 8140 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:59:51.780074 8140 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:59:51.781617 8140 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:59:51.792687 8140 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9fa46e0fbb384d4c92aebae63e956734: Generated new TSK 0
I20260806 02:59:51.793588 8140 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.74.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.193.keytab.
Entry for principal kudu/127.31.74.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.193.keytab.
I20260806 02:59:51.871733 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.193:0
--local_ip_for_outbound_sockets=127.31.74.193
--webserver_interface=127.31.74.193
--webserver_port=0
--tserver_master_addrs=127.31.74.254:45993
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.193.keytab
--principal=kudu/127.31.74.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:45635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:59:52.276875 8152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:59:52.277189 8152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:59:52.277351 8152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:59:52.287941 8152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:59:52.288168 8152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:59:52.288580 8152 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.193
I20260806 02:59:52.300956 8152 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:45635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.193.keytab
--principal=kudu/127.31.74.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.74.193
--webserver_port=0
--tserver_master_addrs=127.31.74.254:45993
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.193
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:59:52.302452 8152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:59:52.303859 8152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:59:52.319901 8157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:52.320102 8160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:52.320129 8158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:59:52.321532 8152 server_base.cc:1034] running on GCE node
I20260806 02:59:52.322295 8152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:59:52.323428 8152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:59:52.324764 8152 hybrid_clock.cc:648] HybridClock initialized: now 1785985192324664 us; error 70 us; skew 500 ppm
I20260806 02:59:52.325160 8152 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:59:52 dist-test-slave-dhph krb5kdc[8044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985192, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:59:52.329700 8152 init.cc:377] Logged in from keytab as kudu/127.31.74.193@KRBTEST.COM (short username kudu)
I20260806 02:59:52.331794 8152 webserver.cc:460] Webserver started at http://127.31.74.193:40091/ using document root <none> and password file <none>
I20260806 02:59:52.332592 8152 fs_manager.cc:359] Metadata directory not provided
I20260806 02:59:52.332746 8152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:59:52.333076 8152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:59:52.336685 8152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-0/data/instance:
uuid: "a5a2d5330342485bb87220f5ff3ad649"
format_stamp: "Formatted at 2026-08-06 02:59:52 on dist-test-slave-dhph"
server_key: "769aa19d825cd74c639fb1b91f5541cc"
server_key_iv: "4bf162a2fcd3dd50cd1c923de7011bdf"
server_key_version: "encryptionkey@0"
I20260806 02:59:52.337723 8152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-0/wal/instance:
uuid: "a5a2d5330342485bb87220f5ff3ad649"
format_stamp: "Formatted at 2026-08-06 02:59:52 on dist-test-slave-dhph"
server_key: "769aa19d825cd74c639fb1b91f5541cc"
server_key_iv: "4bf162a2fcd3dd50cd1c923de7011bdf"
server_key_version: "encryptionkey@0"
I20260806 02:59:52.344264 8152 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260806 02:59:52.348716 8167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:52.350229 8152 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 02:59:52.350474 8152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-0/wal
uuid: "a5a2d5330342485bb87220f5ff3ad649"
format_stamp: "Formatted at 2026-08-06 02:59:52 on dist-test-slave-dhph"
server_key: "769aa19d825cd74c639fb1b91f5541cc"
server_key_iv: "4bf162a2fcd3dd50cd1c923de7011bdf"
server_key_version: "encryptionkey@0"
I20260806 02:59:52.350764 8152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:59:52.363659 8152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:59:52.369349 8152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:59:52.369700 8152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:59:52.370926 8152 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:59:52.372488 8152 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:59:52.372612 8152 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:52.372782 8152 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:59:52.372864 8152 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:52.418000 8152 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.193:38781
I20260806 02:59:52.418006 8280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.193:38781 every 8 connection(s)
I20260806 02:59:52.419627 8152 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
I20260806 02:59:52.427783 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 8152
I20260806 02:59:52.428211 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-0/wal/instance
I20260806 02:59:52.428982 32043 external_mini_cluster.cc:1428] Setting key 5cb08bb7a876fd6649b59b93357f6be6
Aug 06 02:59:52 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985192, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.193@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 02:59:52.444500 8283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:52.423304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.193:43807 (local address 127.31.74.254:45993)
0806 02:59:52.423796 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:52.423801 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:52.425047 (+ 1246us) server_negotiation.cc:407] Connection header received
0806 02:59:52.427330 (+ 2283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:52.427335 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:52.427443 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:52.427571 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:52.429794 (+ 2223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:52.430756 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:52.432545 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:52.433005 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:52.433170 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:52.437587 (+ 4417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:52.437594 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:52.437598 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:52.437639 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:52.440985 (+ 3346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:52.441903 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:52.441911 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:52.441920 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:52.442050 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:52.442528 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:52.442536 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:52.442539 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:52.442901 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:52.443088 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:52.443861 (+ 773us) server_negotiation.cc:299] Negotiation successful
0806 02:59:52.444128 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":269,"threads_started":1}
I20260806 02:59:52.445571 8284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:52.423986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:45993 (local address 127.31.74.193:43807)
0806 02:59:52.424776 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:52.424824 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:52.427010 (+ 2186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:52.427837 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:52.427847 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:52.428523 (+ 676us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:59:52.429562 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:52.429586 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:52.430927 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:52.430942 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:52.432351 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:52.432372 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:52.433285 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:52.433296 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:52.433583 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:52.434601 (+ 1018us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:59:52.434685 (+ 84us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:59:52.437280 (+ 2595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:59:52.441219 (+ 3939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:52.441231 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:52.441253 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:52.441757 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:52.442185 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:52.442189 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:52.442193 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:52.442404 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:52.443042 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:59:52.443051 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:59:52.443449 (+ 398us) client_negotiation.cc:769] Sending connection context
0806 02:59:52.443745 (+ 296us) client_negotiation.cc:240] Negotiation successful
0806 02:59:52.444339 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":328,"threads_started":1}
I20260806 02:59:52.447348 8281 heartbeater.cc:347] Connected to a master server at 127.31.74.254:45993
I20260806 02:59:52.447674 8281 heartbeater.cc:464] Registering TS with master...
I20260806 02:59:52.448455 8281 heartbeater.cc:511] Master 127.31.74.254:45993 requested a full tablet report, sending...
I20260806 02:59:52.451136 8090 ts_manager.cc:194] Registered new tserver with Master: a5a2d5330342485bb87220f5ff3ad649 (127.31.74.193:38781)
I20260806 02:59:52.452772 8090 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.74.193@KRBTEST.COM'} at 127.31.74.193:43807
WARNING: no policy specified for kudu/127.31.74.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.194.keytab.
Entry for principal kudu/127.31.74.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.194.keytab.
I20260806 02:59:52.508154 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.194:0
--local_ip_for_outbound_sockets=127.31.74.194
--webserver_interface=127.31.74.194
--webserver_port=0
--tserver_master_addrs=127.31.74.254:45993
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.194.keytab
--principal=kudu/127.31.74.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:45635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:59:52.904811 8288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:59:52.905125 8288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:59:52.905292 8288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:59:52.913417 8288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:59:52.913555 8288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:59:52.913745 8288 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.194
I20260806 02:59:52.924628 8288 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:45635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.194.keytab
--principal=kudu/127.31.74.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.74.194
--webserver_port=0
--tserver_master_addrs=127.31.74.254:45993
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.194
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:59:52.926059 8288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:59:52.927534 8288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:59:52.941504 8293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:52.941749 8294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:52.942184 8296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:59:52.943239 8288 server_base.cc:1034] running on GCE node
I20260806 02:59:52.943931 8288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:59:52.945350 8288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:59:52.946686 8288 hybrid_clock.cc:648] HybridClock initialized: now 1785985192946562 us; error 59 us; skew 500 ppm
I20260806 02:59:52.947080 8288 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:59:52 dist-test-slave-dhph krb5kdc[8044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985192, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:59:52.950681 8288 init.cc:377] Logged in from keytab as kudu/127.31.74.194@KRBTEST.COM (short username kudu)
I20260806 02:59:52.952503 8288 webserver.cc:460] Webserver started at http://127.31.74.194:34893/ using document root <none> and password file <none>
I20260806 02:59:52.953244 8288 fs_manager.cc:359] Metadata directory not provided
I20260806 02:59:52.953357 8288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:59:52.953620 8288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:59:52.956993 8288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-1/data/instance:
uuid: "23d3a9953fd64c20a4abffc227474844"
format_stamp: "Formatted at 2026-08-06 02:59:52 on dist-test-slave-dhph"
server_key: "d0ea518d656a9836b04a9c37b0822047"
server_key_iv: "7739dc6a4e6ae97d3bd8d99c0a14d3dd"
server_key_version: "encryptionkey@0"
I20260806 02:59:52.957777 8288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-1/wal/instance:
uuid: "23d3a9953fd64c20a4abffc227474844"
format_stamp: "Formatted at 2026-08-06 02:59:52 on dist-test-slave-dhph"
server_key: "d0ea518d656a9836b04a9c37b0822047"
server_key_iv: "7739dc6a4e6ae97d3bd8d99c0a14d3dd"
server_key_version: "encryptionkey@0"
I20260806 02:59:52.963718 8288 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260806 02:59:52.968147 8304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:52.969732 8288 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260806 02:59:52.969897 8288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-1/wal
uuid: "23d3a9953fd64c20a4abffc227474844"
format_stamp: "Formatted at 2026-08-06 02:59:52 on dist-test-slave-dhph"
server_key: "d0ea518d656a9836b04a9c37b0822047"
server_key_iv: "7739dc6a4e6ae97d3bd8d99c0a14d3dd"
server_key_version: "encryptionkey@0"
I20260806 02:59:52.970098 8288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:59:52.991885 8288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:59:52.997027 8288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:59:52.997298 8288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:59:52.998288 8288 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:59:52.999847 8288 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:59:52.999930 8288 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:53.000021 8288 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:59:53.000066 8288 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:53.041672 8288 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.194:36887
I20260806 02:59:53.041728 8419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.194:36887 every 8 connection(s)
I20260806 02:59:53.043092 8288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
I20260806 02:59:53.045075 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 8288
I20260806 02:59:53.045256 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-1/wal/instance
I20260806 02:59:53.045756 32043 external_mini_cluster.cc:1428] Setting key fac07ba74f40b21c9a60b61d9aa80a6d
Aug 06 02:59:53 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985192, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.194@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 02:59:53.067802 8422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:53.045829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.194:33651 (local address 127.31.74.254:45993)
0806 02:59:53.046327 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:53.046334 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:53.047544 (+ 1210us) server_negotiation.cc:407] Connection header received
0806 02:59:53.049452 (+ 1908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:53.049460 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:53.049565 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:53.049693 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:53.051595 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:53.052727 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:53.054919 (+ 2192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:53.055883 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:53.056056 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:53.060690 (+ 4634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:53.060696 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:53.060699 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:53.060739 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:53.063814 (+ 3075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:53.064961 (+ 1147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:53.064966 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:53.064968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:53.065098 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:53.065696 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:53.065700 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:53.065702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:53.066011 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:53.066145 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:53.067280 (+ 1135us) server_negotiation.cc:299] Negotiation successful
0806 02:59:53.067472 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":240,"threads_started":1}
I20260806 02:59:53.069592 8423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:53.046471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:45993 (local address 127.31.74.194:33651)
0806 02:59:53.047375 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:53.047438 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:53.049164 (+ 1726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:53.049896 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:53.049904 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:53.050458 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:59:53.051225 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:53.051251 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:53.052908 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:53.052912 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:53.054750 (+ 1838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:53.054765 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:53.056181 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:53.056188 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:53.056398 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:53.057234 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:59:53.057267 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:59:53.060360 (+ 3093us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:59:53.063996 (+ 3636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:53.064009 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:53.064033 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:53.064776 (+ 743us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:53.065226 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:53.065235 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:53.065242 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:53.065532 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:53.066169 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:59:53.066182 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:59:53.066788 (+ 606us) client_negotiation.cc:769] Sending connection context
0806 02:59:53.067153 (+ 365us) client_negotiation.cc:240] Negotiation successful
0806 02:59:53.067865 (+ 712us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":381,"threads_started":1}
I20260806 02:59:53.071842 8420 heartbeater.cc:347] Connected to a master server at 127.31.74.254:45993
I20260806 02:59:53.072224 8420 heartbeater.cc:464] Registering TS with master...
I20260806 02:59:53.073331 8420 heartbeater.cc:511] Master 127.31.74.254:45993 requested a full tablet report, sending...
I20260806 02:59:53.075233 8090 ts_manager.cc:194] Registered new tserver with Master: 23d3a9953fd64c20a4abffc227474844 (127.31.74.194:36887)
I20260806 02:59:53.076131 8090 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.74.194@KRBTEST.COM'} at 127.31.74.194:33651
WARNING: no policy specified for kudu/127.31.74.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.195.keytab.
Entry for principal kudu/127.31.74.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.195.keytab.
I20260806 02:59:53.119263 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.195:0
--local_ip_for_outbound_sockets=127.31.74.195
--webserver_interface=127.31.74.195
--webserver_port=0
--tserver_master_addrs=127.31.74.254:45993
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.195.keytab
--principal=kudu/127.31.74.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:45635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:59:53.456858 8281 heartbeater.cc:503] Master 127.31.74.254:45993 was elected leader, sending a full tablet report...
W20260806 02:59:53.509877 8426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:59:53.510201 8426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:59:53.510305 8426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:59:53.519147 8426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:59:53.519282 8426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:59:53.519411 8426 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.195
I20260806 02:59:53.530118 8426 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:45635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.195.keytab
--principal=kudu/127.31.74.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.74.195
--webserver_port=0
--tserver_master_addrs=127.31.74.254:45993
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.195
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:59:53.531725 8426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:59:53.533202 8426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:59:53.548525 8431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:53.548468 8432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:53.548542 8434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:59:53.549589 8426 server_base.cc:1034] running on GCE node
I20260806 02:59:53.550320 8426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:59:53.551467 8426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:59:53.552727 8426 hybrid_clock.cc:648] HybridClock initialized: now 1785985193552660 us; error 60 us; skew 500 ppm
I20260806 02:59:53.553133 8426 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:59:53 dist-test-slave-dhph krb5kdc[8044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985193, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:59:53.556905 8426 init.cc:377] Logged in from keytab as kudu/127.31.74.195@KRBTEST.COM (short username kudu)
I20260806 02:59:53.561470 8426 webserver.cc:460] Webserver started at http://127.31.74.195:36959/ using document root <none> and password file <none>
I20260806 02:59:53.562356 8426 fs_manager.cc:359] Metadata directory not provided
I20260806 02:59:53.562528 8426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:59:53.562901 8426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:59:53.566984 8426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-2/data/instance:
uuid: "3ead412dcf5d496ba6fe247d322ca325"
format_stamp: "Formatted at 2026-08-06 02:59:53 on dist-test-slave-dhph"
server_key: "cf14287147c83bc29e71bd3e0f307818"
server_key_iv: "770f82b00e019c5b935127abc7f3897c"
server_key_version: "encryptionkey@0"
I20260806 02:59:53.567898 8426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-2/wal/instance:
uuid: "3ead412dcf5d496ba6fe247d322ca325"
format_stamp: "Formatted at 2026-08-06 02:59:53 on dist-test-slave-dhph"
server_key: "cf14287147c83bc29e71bd3e0f307818"
server_key_iv: "770f82b00e019c5b935127abc7f3897c"
server_key_version: "encryptionkey@0"
I20260806 02:59:53.573836 8426 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 02:59:53.578418 8441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:53.580022 8426 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260806 02:59:53.580225 8426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-2/wal
uuid: "3ead412dcf5d496ba6fe247d322ca325"
format_stamp: "Formatted at 2026-08-06 02:59:53 on dist-test-slave-dhph"
server_key: "cf14287147c83bc29e71bd3e0f307818"
server_key_iv: "770f82b00e019c5b935127abc7f3897c"
server_key_version: "encryptionkey@0"
I20260806 02:59:53.580462 8426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:59:53.658918 8426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:59:53.665619 8426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:59:53.666003 8426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:59:53.667229 8426 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:59:53.668860 8426 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:59:53.669064 8426 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:53.669307 8426 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:59:53.669427 8426 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:53.713372 8426 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.195:39173
I20260806 02:59:53.713510 8556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.195:39173 every 8 connection(s)
I20260806 02:59:53.715034 8426 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
I20260806 02:59:53.715934 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 8426
I20260806 02:59:53.716166 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-2/wal/instance
I20260806 02:59:53.716780 32043 external_mini_cluster.cc:1428] Setting key e53e025b6de211e8b45b9714251a5232
Aug 06 02:59:53 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985193, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.195@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 02:59:53.742833 8559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:53.718202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.195:44733 (local address 127.31.74.254:45993)
0806 02:59:53.718792 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:53.718798 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:53.720273 (+ 1475us) server_negotiation.cc:407] Connection header received
0806 02:59:53.723387 (+ 3114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:53.723393 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:53.723498 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:53.723620 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:53.725339 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:53.726281 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:53.728202 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:53.728671 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:53.728850 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:53.733822 (+ 4972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:53.733828 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:53.733831 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:53.733874 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:53.737417 (+ 3543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:53.739622 (+ 2205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:53.739630 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:53.739633 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:53.739812 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:53.740455 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:53.740459 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:53.740462 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:53.740938 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:53.741089 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:53.742077 (+ 988us) server_negotiation.cc:299] Negotiation successful
0806 02:59:53.742310 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":299,"threads_started":1}
I20260806 02:59:53.743655 8560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:53.719045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:45993 (local address 127.31.74.195:44733)
0806 02:59:53.720091 (+ 1046us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:53.720142 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:53.723066 (+ 2924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:53.723889 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:53.723897 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:53.724452 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:59:53.725051 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:53.725069 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:53.726496 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:53.726501 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:53.728028 (+ 1527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:53.728048 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:53.728959 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:53.728967 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:53.729236 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:53.730192 (+ 956us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:59:53.730226 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:59:53.733534 (+ 3308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:59:53.737638 (+ 4104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:53.737652 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:53.737675 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:53.739138 (+ 1463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:53.740000 (+ 862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:59:53.740004 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:59:53.740007 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:59:53.740269 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:59:53.741121 (+ 852us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:59:53.741132 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:59:53.741637 (+ 505us) client_negotiation.cc:769] Sending connection context
0806 02:59:53.741891 (+ 254us) client_negotiation.cc:240] Negotiation successful
0806 02:59:53.742331 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":754,"thread_start_us":463,"threads_started":1}
I20260806 02:59:53.745760 8557 heartbeater.cc:347] Connected to a master server at 127.31.74.254:45993
I20260806 02:59:53.746229 8557 heartbeater.cc:464] Registering TS with master...
I20260806 02:59:53.747184 8557 heartbeater.cc:511] Master 127.31.74.254:45993 requested a full tablet report, sending...
I20260806 02:59:53.749230 8090 ts_manager.cc:194] Registered new tserver with Master: 3ead412dcf5d496ba6fe247d322ca325 (127.31.74.195:39173)
I20260806 02:59:53.750198 8090 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.74.195@KRBTEST.COM'} at 127.31.74.195:44733
I20260806 02:59:53.763497 32043 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.31.74.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.74.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.74.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/krb5kdc/oryx_127.31.74.193.keytab.
Entry for principal oryx/127.31.74.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/krb5kdc/oryx_127.31.74.193.keytab.
WARNING: no policy specified for oryx/127.31.74.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.74.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.74.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/krb5kdc/oryx_127.31.74.194.keytab.
Entry for principal oryx/127.31.74.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/krb5kdc/oryx_127.31.74.194.keytab.
WARNING: no policy specified for oryx/127.31.74.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.74.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.74.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/krb5kdc/oryx_127.31.74.195.keytab.
Entry for principal oryx/127.31.74.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/krb5kdc/oryx_127.31.74.195.keytab.
I20260806 02:59:53.987046 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 8152
Aug 06 02:59:53 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.74.193@KRBTEST.COM
I20260806 02:59:54.006402 8568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:53.988552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49754 (local address 127.31.74.193:38781)
0806 02:59:53.989518 (+ 966us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:53.989526 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:53.989558 (+ 32us) server_negotiation.cc:407] Connection header received
0806 02:59:53.989632 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:53.989639 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:53.989869 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:53.990026 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:53.991737 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:53.992826 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:53.994093 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:53.994794 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:53.994902 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:53.999565 (+ 4663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:53.999581 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:53.999599 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:53.999649 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:54.003301 (+ 3652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:54.004084 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:54.004093 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:54.004095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:54.004238 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:54.004640 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:54.004645 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:54.004648 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:54.005043 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:54.005363 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:54.005677 (+ 314us) server_negotiation.cc:299] Negotiation successful
0806 02:59:54.005999 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":766,"thread_start_us":571,"threads_started":1}
I20260806 02:59:54.007779 8276 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:59:54.079130 8420 heartbeater.cc:503] Master 127.31.74.254:45993 was elected leader, sending a full tablet report...
I20260806 02:59:54.101178 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 8152
I20260806 02:59:54.123113 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 8288
Aug 06 02:59:54 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.74.194@KRBTEST.COM
I20260806 02:59:54.143646 8570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:54.124398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57600 (local address 127.31.74.194:36887)
0806 02:59:54.125056 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:54.125063 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:54.125113 (+ 50us) server_negotiation.cc:407] Connection header received
0806 02:59:54.125177 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:54.125183 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:54.125404 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:54.125539 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:54.127399 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:54.128610 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:54.130519 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:54.131286 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:54.131414 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:54.136597 (+ 5183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:54.136616 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:54.136634 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:54.136683 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:54.140496 (+ 3813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:54.141321 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:54.141330 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:54.141333 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:54.141481 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:54.141947 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:54.141951 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:54.141953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:54.142323 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:54.142570 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:54.142990 (+ 420us) server_negotiation.cc:299] Negotiation successful
0806 02:59:54.143205 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":318,"threads_started":1}
I20260806 02:59:54.145011 8415 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:59:54.234997 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 8288
I20260806 02:59:54.254918 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 8426
Aug 06 02:59:54 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.74.195@KRBTEST.COM
I20260806 02:59:54.274713 8572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:54.255991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60736 (local address 127.31.74.195:39173)
0806 02:59:54.256677 (+ 686us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:54.256685 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:54.256712 (+ 27us) server_negotiation.cc:407] Connection header received
0806 02:59:54.256786 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:54.256792 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:54.257017 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:54.257171 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:54.258633 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:54.259733 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:54.261364 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:54.262009 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:54.262150 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:54.267769 (+ 5619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:54.267786 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:54.267803 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:54.267846 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:54.271161 (+ 3315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:54.271996 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:54.272005 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:54.272007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:54.272155 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:54.272641 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:54.272646 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:54.272648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:54.273056 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:54.273335 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:54.273741 (+ 406us) server_negotiation.cc:299] Negotiation successful
0806 02:59:54.273980 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":381,"threads_started":1}
I20260806 02:59:54.276305 8552 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:59:54.367578 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 8426
I20260806 02:59:54.388387 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 8058
I20260806 02:59:54.389685 8120 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:59:54.504745 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 8058
I20260806 02:59:54.529891 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:45993
--webserver_interface=127.31.74.254
--webserver_port=41489
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab
--principal=kudu/127.31.74.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:45635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.74.254:45993
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:59:54.926019 8575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:59:54.926385 8575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:59:54.926484 8575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:59:54.937798 8575 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:59:54.937959 8575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:59:54.938011 8575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:59:54.938066 8575 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:59:54.938107 8575 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:59:54.951361 8575 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:45635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.74.254:45993
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab
--principal=kudu/127.31.74.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:45993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.74.254
--webserver_port=41489
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:59:54.952993 8575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:59:54.954737 8575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:59:54.970822 8580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:54.970785 8583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:54.970753 8581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:59:54.972271 8575 server_base.cc:1034] running on GCE node
I20260806 02:59:54.973652 8575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:59:54.975683 8575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:59:54.977020 8575 hybrid_clock.cc:648] HybridClock initialized: now 1785985194976949 us; error 88 us; skew 500 ppm
I20260806 02:59:54.977579 8575 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:59:54 dist-test-slave-dhph krb5kdc[8044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985194, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:59:54.989809 8575 init.cc:377] Logged in from keytab as kudu/127.31.74.254@KRBTEST.COM (short username kudu)
I20260806 02:59:54.992138 8575 webserver.cc:460] Webserver started at http://127.31.74.254:41489/ using document root <none> and password file <none>
I20260806 02:59:54.993165 8575 fs_manager.cc:359] Metadata directory not provided
I20260806 02:59:54.993427 8575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:59:55.001111 8575 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 02:59:55.004804 8591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:55.006757 8575 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 02:59:55.006942 8575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/master-0/wal
uuid: "9fa46e0fbb384d4c92aebae63e956734"
format_stamp: "Formatted at 2026-08-06 02:59:51 on dist-test-slave-dhph"
server_key: "67c197232d5de389895e3210224a5081"
server_key_iv: "03cdf1699ba40659ae59dbfda4d558fe"
server_key_version: "encryptionkey@0"
I20260806 02:59:55.007989 8575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:59:55.031092 8575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:59:55.037284 8575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:59:55.037540 8575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:59:55.064649 8575 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.254:45993
I20260806 02:59:55.064733 8643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.254:45993 every 8 connection(s)
I20260806 02:59:55.066527 8575 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
I20260806 02:59:55.067875 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 8575
I20260806 02:59:55.068961 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.193:38781
--local_ip_for_outbound_sockets=127.31.74.193
--tserver_master_addrs=127.31.74.254:45993
--webserver_port=40091
--webserver_interface=127.31.74.193
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.193.keytab
--principal=kudu/127.31.74.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:45635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.31.74.193
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/krb5kdc/oryx_127.31.74.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:59:55.077802 8644 sys_catalog.cc:263] Verifying existing consensus state
I20260806 02:59:55.082834 8644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fa46e0fbb384d4c92aebae63e956734: Bootstrap starting.
I20260806 02:59:55.104383 8644 log.cc:824] T 00000000000000000000000000000000 P 9fa46e0fbb384d4c92aebae63e956734: Log is configured to *not* fsync() on all Append() calls
I20260806 02:59:55.117864 8644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fa46e0fbb384d4c92aebae63e956734: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 02:59:55.119385 8644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fa46e0fbb384d4c92aebae63e956734: Bootstrap complete.
I20260806 02:59:55.134219 8644 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9fa46e0fbb384d4c92aebae63e956734 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fa46e0fbb384d4c92aebae63e956734" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 45993 } }
I20260806 02:59:55.135819 8644 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9fa46e0fbb384d4c92aebae63e956734 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fa46e0fbb384d4c92aebae63e956734, State: Initialized, Role: FOLLOWER
I20260806 02:59:55.136552 8644 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9fa46e0fbb384d4c92aebae63e956734 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fa46e0fbb384d4c92aebae63e956734" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 45993 } }
I20260806 02:59:55.136978 8644 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9fa46e0fbb384d4c92aebae63e956734 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:59:55.137501 8644 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9fa46e0fbb384d4c92aebae63e956734 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:59:55.137759 8644 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9fa46e0fbb384d4c92aebae63e956734 [term 1 FOLLOWER]: Advancing to term 2
I20260806 02:59:55.142421 8644 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9fa46e0fbb384d4c92aebae63e956734 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fa46e0fbb384d4c92aebae63e956734" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 45993 } }
I20260806 02:59:55.143052 8644 leader_election.cc:302] T 00000000000000000000000000000000 P 9fa46e0fbb384d4c92aebae63e956734 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9fa46e0fbb384d4c92aebae63e956734; no voters:
I20260806 02:59:55.143797 8644 leader_election.cc:288] T 00000000000000000000000000000000 P 9fa46e0fbb384d4c92aebae63e956734 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 02:59:55.143956 8649 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9fa46e0fbb384d4c92aebae63e956734 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 02:59:55.145292 8649 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9fa46e0fbb384d4c92aebae63e956734 [term 2 LEADER]: Becoming Leader. State: Replica: 9fa46e0fbb384d4c92aebae63e956734, State: Running, Role: LEADER
I20260806 02:59:55.146030 8649 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9fa46e0fbb384d4c92aebae63e956734 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fa46e0fbb384d4c92aebae63e956734" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 45993 } }
I20260806 02:59:55.146447 8644 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9fa46e0fbb384d4c92aebae63e956734 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:59:55.149650 8651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fa46e0fbb384d4c92aebae63e956734 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9fa46e0fbb384d4c92aebae63e956734. Latest consensus state: current_term: 2 leader_uuid: "9fa46e0fbb384d4c92aebae63e956734" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fa46e0fbb384d4c92aebae63e956734" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 45993 } } }
I20260806 02:59:55.150420 8651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fa46e0fbb384d4c92aebae63e956734 [sys.catalog]: This master's current role is: LEADER
I20260806 02:59:55.149940 8650 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fa46e0fbb384d4c92aebae63e956734 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9fa46e0fbb384d4c92aebae63e956734" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fa46e0fbb384d4c92aebae63e956734" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 45993 } } }
I20260806 02:59:55.151105 8650 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fa46e0fbb384d4c92aebae63e956734 [sys.catalog]: This master's current role is: LEADER
I20260806 02:59:55.164796 8656 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:59:55.176678 8656 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:59:55.184012 8656 catalog_manager.cc:1259] Loaded cluster ID: 47947b18a85e4961b5404e2791103b41
I20260806 02:59:55.184279 8656 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:59:55.195873 8656 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:59:55.199442 8656 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9fa46e0fbb384d4c92aebae63e956734: Loaded TSK: 0
I20260806 02:59:55.201354 8656 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 02:59:55.530726 8647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:59:55.531112 8647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:59:55.531291 8647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:59:55.540031 8647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:59:55.540232 8647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:59:55.540421 8647 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.193
I20260806 02:59:55.552451 8647 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:45635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/krb5kdc/oryx_127.31.74.193.keytab
--principal=oryx/127.31.74.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.193:38781
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.74.193
--webserver_port=40091
--tserver_master_addrs=127.31.74.254:45993
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.193
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:59:55.554042 8647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:59:55.555724 8647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:59:55.575007 8671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:55.575053 8672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:55.575796 8674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:59:55.576629 8647 server_base.cc:1034] running on GCE node
I20260806 02:59:55.577560 8647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:59:55.579044 8647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:59:55.580332 8647 hybrid_clock.cc:648] HybridClock initialized: now 1785985195580275 us; error 40 us; skew 500 ppm
I20260806 02:59:55.580785 8647 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:59:55 dist-test-slave-dhph krb5kdc[8044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985195, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.74.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:59:55.585819 8647 init.cc:377] Logged in from keytab as oryx/127.31.74.193@KRBTEST.COM (short username oryx)
I20260806 02:59:55.588505 8647 webserver.cc:460] Webserver started at http://127.31.74.193:40091/ using document root <none> and password file <none>
I20260806 02:59:55.589454 8647 fs_manager.cc:359] Metadata directory not provided
I20260806 02:59:55.589643 8647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:59:55.596967 8647 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 02:59:55.602955 8681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:55.606029 8647 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.003s sys 0.001s
I20260806 02:59:55.606673 8647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-0/wal
uuid: "a5a2d5330342485bb87220f5ff3ad649"
format_stamp: "Formatted at 2026-08-06 02:59:52 on dist-test-slave-dhph"
server_key: "769aa19d825cd74c639fb1b91f5541cc"
server_key_iv: "4bf162a2fcd3dd50cd1c923de7011bdf"
server_key_version: "encryptionkey@0"
I20260806 02:59:55.608453 8647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:59:55.653836 8647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:59:55.663661 8647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:59:55.664116 8647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:59:55.665516 8647 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:59:55.669472 8647 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:59:55.669687 8647 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:55.669836 8647 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:59:55.669945 8647 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:55.721042 8647 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.193:38781
I20260806 02:59:55.721148 8795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.193:38781 every 8 connection(s)
I20260806 02:59:55.722888 8647 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
I20260806 02:59:55.733672 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 8647
I20260806 02:59:55.734355 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.194:36887
--local_ip_for_outbound_sockets=127.31.74.194
--tserver_master_addrs=127.31.74.254:45993
--webserver_port=34893
--webserver_interface=127.31.74.194
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.194.keytab
--principal=kudu/127.31.74.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:45635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.31.74.194
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/krb5kdc/oryx_127.31.74.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 02:59:55 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.193@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:59:55 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.193@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
W20260806 02:59:56.239583 8800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:59:56.239915 8800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:59:56.240087 8800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:59:56.248658 8800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:59:56.249060 8800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:59:56.249279 8800 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.194
I20260806 02:59:56.262035 8800 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:45635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/krb5kdc/oryx_127.31.74.194.keytab
--principal=oryx/127.31.74.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.194:36887
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.74.194
--webserver_port=34893
--tserver_master_addrs=127.31.74.254:45993
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.194
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:59:56.263602 8800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:59:56.264991 8800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:59:56.287580 8807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:56.287617 8806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:56.288172 8809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:59:56.290659 8800 server_base.cc:1034] running on GCE node
I20260806 02:59:56.291635 8800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:59:56.292951 8800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:59:56.294251 8800 hybrid_clock.cc:648] HybridClock initialized: now 1785985196294169 us; error 74 us; skew 500 ppm
I20260806 02:59:56.294916 8800 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:59:56 dist-test-slave-dhph krb5kdc[8044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985196, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.74.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:59:56.299180 8800 init.cc:377] Logged in from keytab as oryx/127.31.74.194@KRBTEST.COM (short username oryx)
I20260806 02:59:56.301934 8800 webserver.cc:460] Webserver started at http://127.31.74.194:34893/ using document root <none> and password file <none>
I20260806 02:59:56.302934 8800 fs_manager.cc:359] Metadata directory not provided
I20260806 02:59:56.303112 8800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:59:56.311547 8800 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.007s sys 0.000s
I20260806 02:59:56.315611 8816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:56.317404 8800 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260806 02:59:56.317577 8800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-1/wal
uuid: "23d3a9953fd64c20a4abffc227474844"
format_stamp: "Formatted at 2026-08-06 02:59:52 on dist-test-slave-dhph"
server_key: "d0ea518d656a9836b04a9c37b0822047"
server_key_iv: "7739dc6a4e6ae97d3bd8d99c0a14d3dd"
server_key_version: "encryptionkey@0"
I20260806 02:59:56.318953 8800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:59:56.347926 8800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:59:56.355626 8800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:59:56.356003 8800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:59:56.357972 8800 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:59:56.359834 8800 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:59:56.359935 8800 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:56.360090 8800 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:59:56.360169 8800 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:56.418412 8800 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.194:36887
I20260806 02:59:56.418469 8932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.194:36887 every 8 connection(s)
I20260806 02:59:56.420225 8800 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
I20260806 02:59:56.427568 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 8800
I20260806 02:59:56.428685 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.195:39173
--local_ip_for_outbound_sockets=127.31.74.195
--tserver_master_addrs=127.31.74.254:45993
--webserver_port=36959
--webserver_interface=127.31.74.195
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.195.keytab
--principal=kudu/127.31.74.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:45635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.31.74.195
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/krb5kdc/oryx_127.31.74.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 02:59:56 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.194@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:59:56 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.194@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
W20260806 02:59:56.953825 8937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:59:56.954454 8937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:59:56.954885 8937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:59:56.966377 8937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:59:56.966570 8937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:59:56.966845 8937 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.195
I20260806 02:59:56.982371 8937 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:45635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/krb5kdc/oryx_127.31.74.195.keytab
--principal=oryx/127.31.74.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.195:39173
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.74.195
--webserver_port=36959
--tserver_master_addrs=127.31.74.254:45993
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.195
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:59:56.983973 8937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:59:56.985528 8937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:59:57.012161 8943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:59:57.016196 8944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:59:57.018929 8937 server_base.cc:1034] running on GCE node
W20260806 02:59:57.019284 8946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:59:57.020196 8937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:59:57.021559 8937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:59:57.022881 8937 hybrid_clock.cc:648] HybridClock initialized: now 1785985197022798 us; error 65 us; skew 500 ppm
I20260806 02:59:57.023315 8937 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:59:57 dist-test-slave-dhph krb5kdc[8044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985197, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.74.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:59:57.027480 8937 init.cc:377] Logged in from keytab as oryx/127.31.74.195@KRBTEST.COM (short username oryx)
I20260806 02:59:57.029806 8937 webserver.cc:460] Webserver started at http://127.31.74.195:36959/ using document root <none> and password file <none>
I20260806 02:59:57.030802 8937 fs_manager.cc:359] Metadata directory not provided
I20260806 02:59:57.030999 8937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:59:57.037990 8937 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 02:59:57.042356 8953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:57.044482 8937 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 02:59:57.044725 8937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-2/wal
uuid: "3ead412dcf5d496ba6fe247d322ca325"
format_stamp: "Formatted at 2026-08-06 02:59:53 on dist-test-slave-dhph"
server_key: "cf14287147c83bc29e71bd3e0f307818"
server_key_iv: "770f82b00e019c5b935127abc7f3897c"
server_key_version: "encryptionkey@0"
I20260806 02:59:57.045866 8937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:59:57.063382 8937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:59:57.069439 8937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:59:57.070461 8937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:59:57.072606 8937 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:59:57.075057 8937 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:59:57.075203 8937 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:57.075350 8937 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:59:57.075461 8937 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:59:57.129189 8937 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.195:39173
I20260806 02:59:57.129237 9070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.195:39173 every 8 connection(s)
I20260806 02:59:57.131199 8937 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
I20260806 02:59:57.132678 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 8937
I20260806 02:59:57.193442 9073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:57.134019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33046 (local address 127.31.74.254:45993)
0806 02:59:57.147787 (+ 13768us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:57.147797 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:57.148935 (+ 1138us) server_negotiation.cc:407] Connection header received
0806 02:59:57.149041 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:57.149050 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:57.149161 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:57.149277 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:57.153170 (+ 3893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:57.158329 (+ 5159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:57.164189 (+ 5860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:57.165231 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:57.165401 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:57.172933 (+ 7532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:59:57.172956 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:59:57.172992 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:59:57.173065 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:59:57.182385 (+ 9320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:57.184029 (+ 1644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:57.184046 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:57.184063 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:57.184465 (+ 402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:59:57.185197 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:59:57.185207 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:59:57.185212 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:59:57.186240 (+ 1028us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:59:57.186776 (+ 536us) server_negotiation.cc:1035] Waiting for connection context
0806 02:59:57.189130 (+ 2354us) server_negotiation.cc:299] Negotiation successful
0806 02:59:57.190971 (+ 1841us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13528,"thread_start_us":13810,"threads_started":1}
Aug 06 02:59:57 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.195@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:59:57 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.195@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
W20260806 02:59:57.622339 8799 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.885s user 0.001s sys 0.017s
W20260806 02:59:57.625488 8798 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:59:55.726332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.193:48283 (local address 127.31.74.254:45993)
0806 02:59:55.727402 (+ 1070us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:55.727420 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:55.727727 (+ 307us) server_negotiation.cc:407] Connection header received
0806 02:59:55.730002 (+ 2275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:55.730014 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:55.730363 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:55.730760 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:55.732322 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:55.733602 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:55.735405 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:55.736076 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:55.736310 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:57.624840 (+1888530us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.74.193:48283: BlockingRecv error: recv got EOF from 127.31.74.193:48283 (error 108)
Metrics: {"server-negotiator.queue_time_us":745,"thread_start_us":518,"threads_started":1}
I20260806 02:59:57.628032 8799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:55.726469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:45993 (local address 127.31.74.193:48283)
0806 02:59:55.727530 (+ 1061us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:55.727590 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:55.729407 (+ 1817us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:55.730910 (+ 1503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:55.730920 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:55.731568 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:59:55.732108 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:55.732125 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:55.733794 (+ 1669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:55.733798 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:55.735233 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:55.735261 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:55.736359 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:55.736367 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:55.736649 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:55.737454 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:59:55.737479 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:59:57.626172 (+1888693us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.74.254:45993: Server oryx/127.31.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":743,"thread_start_us":554,"threads_started":1}
W20260806 02:59:57.630558 8796 heartbeater.cc:644] Failed to heartbeat to 127.31.74.254:45993 (0 consecutive failures): Not authorized: Failed to ping master at 127.31.74.254:45993: Client connection negotiation failed: client connection to 127.31.74.254:45993: Server oryx/127.31.74.254@KRBTEST.COM not found in Kerberos database
Aug 06 02:59:57 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.193@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:59:57 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.193@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
W20260806 02:59:57.644589 8798 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:59:57.632479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.193:46791 (local address 127.31.74.254:45993)
0806 02:59:57.632664 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:57.632672 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:57.632704 (+ 32us) server_negotiation.cc:407] Connection header received
0806 02:59:57.632883 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:57.632888 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:57.633203 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:57.633376 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:57.634247 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:57.635324 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:57.636812 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:57.637248 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:57.637420 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:57.644300 (+ 6880us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.74.193:46791: BlockingRecv error: recv got EOF from 127.31.74.193:46791 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 02:59:57.645471 8799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:57.631861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:45993 (local address 127.31.74.193:46791)
0806 02:59:57.632453 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:57.632476 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:57.632615 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:57.633405 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:57.633410 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:57.633873 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:59:57.634074 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:57.634083 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:57.635486 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:57.635494 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:57.636590 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:57.636604 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:57.637462 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:57.637468 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:57.637667 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:57.638691 (+ 1024us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:59:57.638748 (+ 57us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:59:57.645213 (+ 6465us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.74.254:45993: Server oryx/127.31.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":423}
Aug 06 02:59:57 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.193@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:59:57 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.193@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:59:57.665488 8799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:57.646357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:45993 (local address 127.31.74.193:50661)
0806 02:59:57.646726 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:57.646751 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:57.647000 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:57.647426 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:57.647434 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:57.647911 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:59:57.648152 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:57.648163 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:57.650425 (+ 2262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:57.650430 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:57.652666 (+ 2236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:57.652686 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:57.653842 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:57.653850 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:57.654053 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:57.655953 (+ 1900us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:59:57.655995 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:59:57.664930 (+ 8935us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.74.254:45993: Server oryx/127.31.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168}
W20260806 02:59:57.665849 8798 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:59:57.646726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.193:50661 (local address 127.31.74.254:45993)
0806 02:59:57.646998 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:57.647005 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:57.647030 (+ 25us) server_negotiation.cc:407] Connection header received
0806 02:59:57.647147 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:57.647154 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:57.647247 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:57.647376 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:57.648336 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:57.650208 (+ 1872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:57.652876 (+ 2668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:57.653514 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:57.653864 (+ 350us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:57.665364 (+ 11500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.74.193:50661: BlockingRecv error: recv got EOF from 127.31.74.193:50661 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260806 02:59:57.666092 8796 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 02:59:58.091763 8936 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.647s user 0.005s sys 0.019s
W20260806 02:59:58.093374 8935 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:59:56.423749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.194:40037 (local address 127.31.74.254:45993)
0806 02:59:56.436369 (+ 12620us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:56.436374 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:56.436400 (+ 26us) server_negotiation.cc:407] Connection header received
0806 02:59:56.436466 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:56.436470 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:56.436556 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:56.436873 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:56.439261 (+ 2388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:56.440221 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:56.442181 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:56.442672 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:56.442817 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:58.093012 (+1650195us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.74.194:40037: BlockingRecv error: recv got EOF from 127.31.74.194:40037 (error 108)
Metrics: {"server-negotiator.queue_time_us":12473,"thread_start_us":12386,"threads_started":1}
I20260806 02:59:58.094041 8936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:56.424928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:45993 (local address 127.31.74.194:40037)
0806 02:59:56.426465 (+ 1537us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:56.426534 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:56.430052 (+ 3518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:56.437575 (+ 7523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:56.437594 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:56.438275 (+ 681us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:59:56.439018 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:56.439040 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:56.440453 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:56.440457 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:56.441997 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:56.442013 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:56.442927 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:56.442937 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:56.443215 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:56.444528 (+ 1313us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:59:56.444553 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:59:58.092786 (+1648233us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.74.254:45993: Server oryx/127.31.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1073,"thread_start_us":735,"threads_started":1}
W20260806 02:59:58.095567 8933 heartbeater.cc:644] Failed to heartbeat to 127.31.74.254:45993 (0 consecutive failures): Not authorized: Failed to ping master at 127.31.74.254:45993: Client connection negotiation failed: client connection to 127.31.74.254:45993: Server oryx/127.31.74.254@KRBTEST.COM not found in Kerberos database
Aug 06 02:59:58 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.194@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:59:58 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.194@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:59:58.113260 8936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:58.097009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:45993 (local address 127.31.74.194:38343)
0806 02:59:58.097378 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:58.097413 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:58.097637 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:58.098122 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:58.098131 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:58.099841 (+ 1710us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:59:58.100240 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:58.100253 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:58.101451 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:58.101460 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:58.103902 (+ 2442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:58.103929 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:58.105158 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:58.105167 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:58.105390 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:58.106465 (+ 1075us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:59:58.106501 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:59:58.112720 (+ 6219us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.74.254:45993: Server oryx/127.31.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106}
W20260806 02:59:58.113427 8935 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:59:58.097068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.194:38343 (local address 127.31.74.254:45993)
0806 02:59:58.097275 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:58.097281 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:58.097661 (+ 380us) server_negotiation.cc:407] Connection header received
0806 02:59:58.097852 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:58.097856 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:58.097925 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:58.098017 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:58.100483 (+ 2466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:58.101257 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:58.104193 (+ 2936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:58.104943 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:58.105088 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:58.113008 (+ 7920us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.74.194:38343: BlockingRecv error: recv got EOF from 127.31.74.194:38343 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 06 02:59:58 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.194@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:59:58 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.194@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:59:58.127028 8936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:58.114268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:45993 (local address 127.31.74.194:33211)
0806 02:59:58.114603 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:58.114630 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:58.114780 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:58.115872 (+ 1092us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:58.115877 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:58.116380 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:59:58.116576 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:58.116586 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:58.118054 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:58.118065 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:58.119617 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:58.119641 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:58.120503 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:58.120510 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:58.120739 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:58.121598 (+ 859us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:59:58.121619 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:59:58.126641 (+ 5022us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.74.254:45993: Server oryx/127.31.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188}
W20260806 02:59:58.127395 8933 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 02:59:58.127620 8935 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:59:58.114993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.194:33211 (local address 127.31.74.254:45993)
0806 02:59:58.115284 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:58.115295 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:58.115323 (+ 28us) server_negotiation.cc:407] Connection header received
0806 02:59:58.115416 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:58.115425 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:58.115570 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:58.115738 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:58.116748 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:58.117883 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:58.119832 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:58.120320 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:58.120479 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:58.126968 (+ 6489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.74.194:33211: BlockingRecv error: recv got EOF from 127.31.74.194:33211 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
W20260806 02:59:58.531101 9076 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.336s user 0.007s sys 0.014s
W20260806 02:59:58.533519 9075 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:59:57.155453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.195:38589 (local address 127.31.74.254:45993)
0806 02:59:57.167331 (+ 11878us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:57.167342 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:57.167382 (+ 40us) server_negotiation.cc:407] Connection header received
0806 02:59:57.175099 (+ 7717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:57.175110 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:57.175308 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:57.175515 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:57.181837 (+ 6322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:57.184162 (+ 2325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:57.188951 (+ 4789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:57.191526 (+ 2575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:57.191725 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:58.532898 (+1341173us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.74.195:38589: BlockingRecv error: recv got EOF from 127.31.74.195:38589 (error 108)
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":11611,"thread_start_us":11656,"threads_started":1}
I20260806 02:59:58.534376 9076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:57.156618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:45993 (local address 127.31.74.195:38589)
0806 02:59:57.163759 (+ 7141us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:57.163850 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:57.173558 (+ 9708us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:57.178026 (+ 4468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:57.178058 (+ 32us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:57.179649 (+ 1591us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:59:57.181333 (+ 1684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:57.181396 (+ 63us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:57.184583 (+ 3187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:57.184594 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:57.188469 (+ 3875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:57.188505 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:57.192573 (+ 4068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:57.192586 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:57.193177 (+ 591us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:57.194840 (+ 1663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:59:57.194873 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:59:58.532449 (+1337576us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.74.254:45993: Server oryx/127.31.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":6454,"thread_start_us":6043,"threads_started":1}
W20260806 02:59:58.535519 9071 heartbeater.cc:644] Failed to heartbeat to 127.31.74.254:45993 (0 consecutive failures): Not authorized: Failed to ping master at 127.31.74.254:45993: Client connection negotiation failed: client connection to 127.31.74.254:45993: Server oryx/127.31.74.254@KRBTEST.COM not found in Kerberos database
Aug 06 02:59:58 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.195@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:59:58 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.195@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:59:58.553431 9076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:58.536578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:45993 (local address 127.31.74.195:50849)
0806 02:59:58.537006 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:58.537033 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:58.537185 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:58.537747 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:58.537753 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:58.538516 (+ 763us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:59:58.539385 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:58.539401 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:58.540641 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:58.540649 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:58.541982 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:58.541996 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:58.543636 (+ 1640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:58.543643 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:58.544021 (+ 378us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:58.545208 (+ 1187us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:59:58.545228 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:59:58.552483 (+ 7255us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.74.254:45993: Server oryx/127.31.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":64}
W20260806 02:59:58.554055 9075 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:59:58.536837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.195:50849 (local address 127.31.74.254:45993)
0806 02:59:58.537125 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:58.537132 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:58.537156 (+ 24us) server_negotiation.cc:407] Connection header received
0806 02:59:58.537496 (+ 340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:58.537503 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:58.537588 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:58.537713 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:58.539587 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:58.540464 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:58.542682 (+ 2218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:58.543310 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:58.543602 (+ 292us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:58.553327 (+ 9725us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.74.195:50849: BlockingRecv error: recv got EOF from 127.31.74.195:50849 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Aug 06 02:59:58 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.195@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:59:58 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.195@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:59:58.567271 9076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:58.554416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:45993 (local address 127.31.74.195:33617)
0806 02:59:58.554831 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:58.554861 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:58.555029 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:58.555667 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:58.555673 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:58.556368 (+ 695us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:59:58.556706 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:58.556719 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:58.558496 (+ 1777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:58.558500 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:58.559696 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:58.559712 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:58.561062 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:58.561069 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:58.561397 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:58.562078 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:59:58.562098 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:59:58.567023 (+ 4925us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.74.254:45993: Server oryx/127.31.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216}
W20260806 02:59:58.567778 9075 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:59:58.554514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.195:33617 (local address 127.31.74.254:45993)
0806 02:59:58.555157 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:58.555165 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:58.555198 (+ 33us) server_negotiation.cc:407] Connection header received
0806 02:59:58.555284 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:58.555291 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:58.555392 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:58.555516 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:58.557145 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:58.558324 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:58.560088 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:58.560588 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:58.560700 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:58.567502 (+ 6802us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.74.195:33617: BlockingRecv error: recv got EOF from 127.31.74.195:33617 (error 108)
Metrics: {"mutex_wait_us":198,"server-negotiator.queue_time_us":423}
W20260806 02:59:58.568073 9071 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 06 02:59:58 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.193@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:59:58 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.193@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:59:58.681653 9078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:58.667053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:45993 (local address 127.31.74.193:56089)
0806 02:59:58.667694 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:58.667716 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:58.668018 (+ 302us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:58.668709 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:58.668713 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:58.669277 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:59:58.669458 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:58.669469 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:58.671133 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:58.671142 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:58.672935 (+ 1793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:58.672954 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:58.674233 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:58.674256 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:58.674415 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:58.676139 (+ 1724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:59:58.676163 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:59:58.681241 (+ 5078us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.74.254:45993: Server oryx/127.31.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":348,"threads_started":1}
W20260806 02:59:58.681772 9075 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:59:58.667158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.193:56089 (local address 127.31.74.254:45993)
0806 02:59:58.667390 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:58.667398 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:58.667815 (+ 417us) server_negotiation.cc:407] Connection header received
0806 02:59:58.668368 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:58.668429 (+ 61us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:58.668527 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:58.668679 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:58.669629 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:58.670849 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:58.673147 (+ 2298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:58.673772 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:58.673904 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:58.681491 (+ 7587us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.74.193:56089: BlockingRecv error: recv got EOF from 127.31.74.193:56089 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Aug 06 02:59:59 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.194@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:59:59 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.194@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:59:59.140172 9079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:59.128489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:45993 (local address 127.31.74.194:38703)
0806 02:59:59.129174 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:59.129198 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:59.129375 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:59.129867 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:59.129872 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:59.130505 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:59:59.130779 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:59.130793 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:59.132072 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:59.132078 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:59.133298 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:59.133313 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:59.134158 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:59.134163 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:59.134317 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:59.135066 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:59:59.135086 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:59:59.139911 (+ 4825us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.74.254:45993: Server oryx/127.31.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":412,"threads_started":1}
W20260806 02:59:59.140936 9075 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:59:59.128794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.194:38703 (local address 127.31.74.254:45993)
0806 02:59:59.129070 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:59.129076 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:59.129555 (+ 479us) server_negotiation.cc:407] Connection header received
0806 02:59:59.129629 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:59.129633 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:59.129699 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:59.129825 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:59.130997 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:59.131900 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:59.133481 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:59.133953 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:59.134064 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:59.140515 (+ 6451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.74.194:38703: BlockingRecv error: recv got EOF from 127.31.74.194:38703 (error 108)
Metrics: {"server-negotiator.queue_time_us":134}
Aug 06 02:59:59 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.195@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:59:59 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.195@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:59:59.581284 9080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:59.568974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:45993 (local address 127.31.74.195:48049)
0806 02:59:59.569721 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:59.569749 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:59.569942 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:59.570891 (+ 949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:59.570897 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:59.571531 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:59:59.571727 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:59.571737 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:59.573205 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:59.573209 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:59.574234 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:59.574248 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:59.575378 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:59.575383 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:59.575518 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:59.576312 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:59:59.576337 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:59:59.581000 (+ 4663us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.74.254:45993: Server oryx/127.31.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":425,"threads_started":1}
W20260806 02:59:59.581676 9075 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:59:59.569210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.195:48049 (local address 127.31.74.254:45993)
0806 02:59:59.569483 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:59.569492 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:59.570130 (+ 638us) server_negotiation.cc:407] Connection header received
0806 02:59:59.570307 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:59.570316 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:59.570652 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:59.570800 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:59.572041 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:59.572989 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:59.574491 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:59.575066 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:59.575192 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:59.581352 (+ 6160us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.74.195:48049: BlockingRecv error: recv got EOF from 127.31.74.195:48049 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
Aug 06 02:59:59 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.193@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:59:59 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.193@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:59:59.695305 9081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:59:59.683103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:45993 (local address 127.31.74.193:60571)
0806 02:59:59.684242 (+ 1139us) negotiation.cc:107] Waiting for socket to connect
0806 02:59:59.684263 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 02:59:59.684558 (+ 295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:59:59.685006 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:59:59.685011 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:59:59.685504 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:59:59.685683 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:59.685692 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:59.686858 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:59.686869 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:59.687823 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:59:59.687836 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:59:59.688809 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:59:59.688816 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:59:59.689006 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:59.689883 (+ 877us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:59:59.689902 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:59:59.694939 (+ 5037us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.74.254:45993: Server oryx/127.31.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":888,"thread_start_us":670,"threads_started":1}
W20260806 02:59:59.695958 9075 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:59:59.683286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.193:60571 (local address 127.31.74.254:45993)
0806 02:59:59.683580 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0806 02:59:59.683587 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:59:59.684495 (+ 908us) server_negotiation.cc:407] Connection header received
0806 02:59:59.684758 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:59:59.684763 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:59:59.684849 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:59:59.684965 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:59:59.685857 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:59.686707 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:59.687984 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:59:59.688631 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:59:59.688772 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:59:59.695649 (+ 6877us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.74.193:60571: BlockingRecv error: recv got EOF from 127.31.74.193:60571 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
Aug 06 03:00:00 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.194@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:00:00 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.194@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260806 03:00:00.158015 9082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:00:00.142059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:45993 (local address 127.31.74.194:50719)
0806 03:00:00.142689 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0806 03:00:00.142707 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 03:00:00.142847 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:00:00.143214 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:00:00.143217 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:00:00.143697 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:00:00.144003 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:00.144018 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:00.145310 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:00.145317 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:00.146757 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:00.146776 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:00.148066 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:00.148073 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:00.148288 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:00.152446 (+ 4158us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:00:00.152480 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:00:00.157713 (+ 5233us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.74.254:45993: Server oryx/127.31.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":348,"threads_started":1}
W20260806 03:00:00.158325 9075 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:00:00.142212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.194:50719 (local address 127.31.74.254:45993)
0806 03:00:00.142562 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0806 03:00:00.142567 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 03:00:00.142882 (+ 315us) server_negotiation.cc:407] Connection header received
0806 03:00:00.142954 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:00:00.142958 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:00:00.143030 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:00:00.143124 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:00:00.144198 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:00.144994 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:00.146968 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:00.147733 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:00.147901 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:00.158027 (+ 10126us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.74.194:50719: BlockingRecv error: recv got EOF from 127.31.74.194:50719 (error 108)
Metrics: {"server-negotiator.queue_time_us":115}
Aug 06 03:00:00 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.195@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:00:00 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.195@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260806 03:00:00.594825 9083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:00:00.582528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:45993 (local address 127.31.74.195:37113)
0806 03:00:00.583257 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0806 03:00:00.583274 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 03:00:00.583398 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:00:00.584078 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:00:00.584086 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:00:00.584896 (+ 810us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:00:00.585115 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:00.585127 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:00.586300 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:00.586310 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:00.587412 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:00.587425 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:00.588228 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:00.588235 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:00.588461 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:00.589250 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:00:00.589269 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:00:00.594405 (+ 5136us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.74.254:45993: Server oryx/127.31.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":344,"threads_started":1}
W20260806 03:00:00.595711 9075 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:00:00.582741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.195:37113 (local address 127.31.74.254:45993)
0806 03:00:00.582977 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0806 03:00:00.582983 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 03:00:00.583502 (+ 519us) server_negotiation.cc:407] Connection header received
0806 03:00:00.583778 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:00:00.583785 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:00:00.583860 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:00:00.584032 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:00:00.585325 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:00.586142 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:00.587583 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:00.588043 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:00.588197 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:00.595245 (+ 7048us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.74.195:37113: BlockingRecv error: recv got EOF from 127.31.74.195:37113 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
Aug 06 03:00:00 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.193@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:00:00 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.193@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260806 03:00:00.708163 9084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:00:00.696868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:45993 (local address 127.31.74.193:58855)
0806 03:00:00.697443 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0806 03:00:00.697461 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 03:00:00.697703 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:00:00.698078 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:00:00.698084 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:00:00.698544 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:00:00.698746 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:00.698755 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:00.699839 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:00.699845 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:00.700828 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:00.700840 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:00.701606 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:00.701612 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:00.701806 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:00.702492 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:00:00.702510 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:00:00.707706 (+ 5196us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.74.254:45993: Server oryx/127.31.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":449,"threads_started":1}
W20260806 03:00:00.708421 9075 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:00:00.697119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.193:58855 (local address 127.31.74.254:45993)
0806 03:00:00.697336 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0806 03:00:00.697343 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 03:00:00.697570 (+ 227us) server_negotiation.cc:407] Connection header received
0806 03:00:00.697867 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:00:00.697872 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:00:00.697934 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:00:00.698038 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:00:00.698959 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:00.699676 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:00.700982 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:00.701441 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:00.701599 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:00.708044 (+ 6445us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.74.193:58855: BlockingRecv error: recv got EOF from 127.31.74.193:58855 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Aug 06 03:00:01 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.194@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:00:01 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.194@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260806 03:00:01.173772 9085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:00:01.159308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:45993 (local address 127.31.74.194:58065)
0806 03:00:01.159994 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0806 03:00:01.160023 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0806 03:00:01.160214 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:00:01.160612 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:00:01.160618 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:00:01.161533 (+ 915us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:00:01.161845 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:01.161869 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:01.163270 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:01.163280 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:01.165199 (+ 1919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:01.165217 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:01.166758 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:01.166777 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:01.167218 (+ 441us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:01.168259 (+ 1041us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:00:01.168474 (+ 215us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:00:01.173382 (+ 4908us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.74.254:45993: Server oryx/127.31.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":357,"threads_started":1}
W20260806 03:00:01.173985 9075 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:00:01.159569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.194:58065 (local address 127.31.74.254:45993)
0806 03:00:01.159939 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0806 03:00:01.159947 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 03:00:01.160125 (+ 178us) server_negotiation.cc:407] Connection header received
0806 03:00:01.160365 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:00:01.160372 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:00:01.160453 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:00:01.160574 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:00:01.162031 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:01.163005 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:01.165469 (+ 2464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:01.166421 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:01.166784 (+ 363us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:01.173683 (+ 6899us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.74.194:58065: BlockingRecv error: recv got EOF from 127.31.74.194:58065 (error 108)
Metrics: {"server-negotiator.queue_time_us":116}
Aug 06 03:00:01 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.195@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:00:01 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.195@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260806 03:00:01.615391 9086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:00:01.596467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:45993 (local address 127.31.74.195:50397)
0806 03:00:01.597086 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0806 03:00:01.597105 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 03:00:01.597295 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:00:01.597699 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:00:01.597705 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:00:01.598143 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:00:01.598353 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:01.598363 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:01.600100 (+ 1737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:01.600108 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:01.603377 (+ 3269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:01.603401 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:01.605054 (+ 1653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:01.605063 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:01.605317 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:01.606683 (+ 1366us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:00:01.606877 (+ 194us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:00:01.615003 (+ 8126us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.74.254:45993: Server oryx/127.31.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":346,"threads_started":1}
W20260806 03:00:01.615559 9075 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:00:01.596487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.195:50397 (local address 127.31.74.254:45993)
0806 03:00:01.596736 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0806 03:00:01.596746 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 03:00:01.597214 (+ 468us) server_negotiation.cc:407] Connection header received
0806 03:00:01.597469 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:00:01.597473 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:00:01.597544 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:00:01.597663 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:00:01.598530 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:01.599791 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:01.603952 (+ 4161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:01.604872 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:01.605018 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:01.615262 (+ 10244us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.74.195:50397: BlockingRecv error: recv got EOF from 127.31.74.195:50397 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Aug 06 03:00:01 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.193@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:00:01 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.193@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260806 03:00:01.726606 9087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:00:01.709480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:45993 (local address 127.31.74.193:48991)
0806 03:00:01.710563 (+ 1083us) negotiation.cc:107] Waiting for socket to connect
0806 03:00:01.710626 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0806 03:00:01.710930 (+ 304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:00:01.711371 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:00:01.711376 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:00:01.712182 (+ 806us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:00:01.712396 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:01.712407 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:01.714368 (+ 1961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:01.714375 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:01.715821 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:01.715857 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:01.716709 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:01.716717 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:01.716934 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:01.718825 (+ 1891us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:00:01.718850 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:00:01.725765 (+ 6915us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.74.254:45993: Server oryx/127.31.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":772,"thread_start_us":570,"threads_started":1}
W20260806 03:00:01.727433 9075 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:00:01.709915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.193:48991 (local address 127.31.74.254:45993)
0806 03:00:01.710122 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0806 03:00:01.710131 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 03:00:01.710727 (+ 596us) server_negotiation.cc:407] Connection header received
0806 03:00:01.711083 (+ 356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:00:01.711092 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:00:01.711185 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:00:01.711313 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:00:01.712581 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:01.714198 (+ 1617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:01.716032 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:01.716526 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:01.716687 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:01.726751 (+ 10064us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.74.193:48991: BlockingRecv error: recv got EOF from 127.31.74.193:48991 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 06 03:00:02 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.194@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:00:02 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.194@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260806 03:00:02.196944 9088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:00:02.175383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:45993 (local address 127.31.74.194:56729)
0806 03:00:02.176295 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0806 03:00:02.176314 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 03:00:02.176525 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:00:02.176947 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:00:02.176952 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:00:02.177452 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:00:02.177641 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:02.177653 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:02.179062 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:02.179070 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:02.180518 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:02.180535 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:02.181403 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:02.181410 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:02.181610 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:02.191029 (+ 9419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:00:02.191078 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:00:02.196625 (+ 5547us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.74.254:45993: Server oryx/127.31.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":772,"thread_start_us":567,"threads_started":1}
W20260806 03:00:02.197609 9075 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:00:02.175590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.194:56729 (local address 127.31.74.254:45993)
0806 03:00:02.176030 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0806 03:00:02.176040 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 03:00:02.176433 (+ 393us) server_negotiation.cc:407] Connection header received
0806 03:00:02.176698 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:00:02.176704 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:00:02.176776 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:00:02.176906 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:00:02.177914 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:02.178895 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:02.180713 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:02.181226 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:02.181384 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:02.197342 (+ 15958us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.74.194:56729: BlockingRecv error: recv got EOF from 127.31.74.194:56729 (error 108)
Metrics: {"server-negotiator.queue_time_us":233}
Aug 06 03:00:02 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.195@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:00:02 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.195@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260806 03:00:02.642463 9089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:00:02.617611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:45993 (local address 127.31.74.195:54567)
0806 03:00:02.618755 (+ 1144us) negotiation.cc:107] Waiting for socket to connect
0806 03:00:02.618776 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 03:00:02.619442 (+ 666us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:00:02.624940 (+ 5498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:00:02.624946 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:00:02.625485 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:00:02.625690 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:02.625702 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:02.628032 (+ 2330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:02.628042 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:02.631438 (+ 3396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:02.631464 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:02.632474 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:02.632479 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:02.632628 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:02.633984 (+ 1356us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:00:02.634156 (+ 172us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:00:02.641941 (+ 7785us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.74.254:45993: Server oryx/127.31.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":937,"thread_start_us":771,"threads_started":1}
W20260806 03:00:02.642736 9075 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:00:02.617595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.195:54567 (local address 127.31.74.254:45993)
0806 03:00:02.618292 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0806 03:00:02.618303 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 03:00:02.619163 (+ 860us) server_negotiation.cc:407] Connection header received
0806 03:00:02.624496 (+ 5333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:00:02.624509 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:00:02.624678 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:00:02.624808 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:00:02.625877 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:02.627427 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:02.631765 (+ 4338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:02.632254 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:02.632386 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:02.642210 (+ 9824us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.74.195:54567: BlockingRecv error: recv got EOF from 127.31.74.195:54567 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Aug 06 03:00:02 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.193@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:00:02 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.193@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260806 03:00:02.744964 9090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:00:02.729275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:45993 (local address 127.31.74.193:58455)
0806 03:00:02.729966 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0806 03:00:02.729988 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 03:00:02.730163 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:00:02.730799 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:00:02.730804 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:00:02.731371 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:00:02.731644 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:02.731659 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:02.732871 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:02.732875 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:02.734036 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:02.734051 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:02.735596 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:02.735605 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:02.735847 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:02.736910 (+ 1063us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:00:02.736954 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:00:02.744599 (+ 7645us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.74.254:45993: Server oryx/127.31.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":405,"threads_started":1}
W20260806 03:00:02.745229 9075 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:00:02.729342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.193:58455 (local address 127.31.74.254:45993)
0806 03:00:02.729709 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0806 03:00:02.729724 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 03:00:02.730260 (+ 536us) server_negotiation.cc:407] Connection header received
0806 03:00:02.730336 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:00:02.730342 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:00:02.730430 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:00:02.730542 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:00:02.731855 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:02.732693 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:02.734328 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:02.735330 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:02.735471 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:02.744932 (+ 9461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.74.193:58455: BlockingRecv error: recv got EOF from 127.31.74.193:58455 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Aug 06 03:00:03 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.194@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:00:03 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.194@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260806 03:00:03.214502 9091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:00:03.198416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:45993 (local address 127.31.74.194:45969)
0806 03:00:03.199327 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0806 03:00:03.199354 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0806 03:00:03.199673 (+ 319us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:00:03.200483 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:00:03.200489 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:00:03.201100 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:00:03.201281 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:03.201292 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:03.203332 (+ 2040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:03.203337 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:03.204515 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:03.204533 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:03.206066 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:03.206072 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:03.206863 (+ 791us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:03.208054 (+ 1191us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:00:03.208080 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:00:03.213816 (+ 5736us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.74.254:45993: Server oryx/127.31.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":347,"threads_started":1}
W20260806 03:00:03.214815 9075 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:00:03.199132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.194:45969 (local address 127.31.74.254:45993)
0806 03:00:03.199423 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0806 03:00:03.199431 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 03:00:03.199455 (+ 24us) server_negotiation.cc:407] Connection header received
0806 03:00:03.199948 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:00:03.199954 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:00:03.200125 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:00:03.200464 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:00:03.201487 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:03.203027 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:03.204801 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:03.205556 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:03.206062 (+ 506us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:03.214130 (+ 8068us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.74.194:45969: BlockingRecv error: recv got EOF from 127.31.74.194:45969 (error 108)
Metrics: {"server-negotiator.queue_time_us":137}
Aug 06 03:00:03 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.195@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:00:03 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.195@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260806 03:00:03.661484 9092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:00:03.644723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:45993 (local address 127.31.74.195:42683)
0806 03:00:03.645578 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0806 03:00:03.645596 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 03:00:03.645783 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:00:03.646977 (+ 1194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:00:03.646982 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:00:03.647581 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:00:03.647760 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:03.647770 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:03.649776 (+ 2006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:03.649782 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:03.651052 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:03.651069 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:03.652611 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:03.652619 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:03.652826 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:03.654125 (+ 1299us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:00:03.654235 (+ 110us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:00:03.661212 (+ 6977us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.74.254:45993: Server oryx/127.31.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":711,"thread_start_us":560,"threads_started":1}
W20260806 03:00:03.661947 9075 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:00:03.644933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.195:42683 (local address 127.31.74.254:45993)
0806 03:00:03.645148 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0806 03:00:03.645155 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 03:00:03.645701 (+ 546us) server_negotiation.cc:407] Connection header received
0806 03:00:03.645923 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:00:03.645930 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:00:03.646757 (+ 827us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:00:03.646933 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:00:03.647951 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:03.649458 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:03.651256 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:03.652394 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:03.652568 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:03.661512 (+ 8944us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.74.195:42683: BlockingRecv error: recv got EOF from 127.31.74.195:42683 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 06 03:00:03 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.193@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:00:03 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.193@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260806 03:00:03.767836 9093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:00:03.748445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:45993 (local address 127.31.74.193:60843)
0806 03:00:03.749504 (+ 1059us) negotiation.cc:107] Waiting for socket to connect
0806 03:00:03.749525 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 03:00:03.749752 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:00:03.750266 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:00:03.750271 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:00:03.750855 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:00:03.751043 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:03.751052 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:03.752572 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:03.752587 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:03.754311 (+ 1724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:03.754343 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:03.755992 (+ 1649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:03.756001 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:03.756250 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:03.757509 (+ 1259us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:00:03.757539 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:00:03.767538 (+ 9999us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.74.254:45993: Server oryx/127.31.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":413,"threads_started":1}
W20260806 03:00:03.768329 9075 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:00:03.748517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.193:60843 (local address 127.31.74.254:45993)
0806 03:00:03.749317 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0806 03:00:03.749322 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 03:00:03.749638 (+ 316us) server_negotiation.cc:407] Connection header received
0806 03:00:03.749922 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:00:03.749927 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:00:03.750001 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:00:03.750095 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:00:03.751278 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:03.752279 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:03.754798 (+ 2519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:03.755689 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:03.755899 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:03.767909 (+ 12010us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.74.193:60843: BlockingRecv error: recv got EOF from 127.31.74.193:60843 (error 108)
Metrics: {"server-negotiator.queue_time_us":133}
Aug 06 03:00:04 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.194@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:00:04 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.194@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260806 03:00:04.233971 9094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:00:04.216461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:45993 (local address 127.31.74.194:50103)
0806 03:00:04.217107 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0806 03:00:04.217127 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 03:00:04.217323 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:00:04.217715 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:00:04.217720 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:00:04.218249 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:00:04.218440 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:04.218450 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:04.220183 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:04.220191 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:04.221303 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:04.221327 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:04.222209 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:04.222217 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:04.222442 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:04.223420 (+ 978us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:00:04.223442 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:00:04.233697 (+ 10255us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.74.254:45993: Server oryx/127.31.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":352,"threads_started":1}
W20260806 03:00:04.234403 9075 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:00:04.216594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.194:50103 (local address 127.31.74.254:45993)
0806 03:00:04.216904 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0806 03:00:04.216912 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 03:00:04.217236 (+ 324us) server_negotiation.cc:407] Connection header received
0806 03:00:04.217475 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:00:04.217480 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:00:04.217548 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:00:04.217670 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:00:04.218960 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:04.219996 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:04.221505 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:04.222011 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:04.222169 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:04.234120 (+ 11951us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.74.194:50103: BlockingRecv error: recv got EOF from 127.31.74.194:50103 (error 108)
Metrics: {"server-negotiator.queue_time_us":123}
Aug 06 03:00:04 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.195@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:00:04 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.195@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260806 03:00:04.677134 9095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:00:04.662872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:45993 (local address 127.31.74.195:56485)
0806 03:00:04.664230 (+ 1358us) negotiation.cc:107] Waiting for socket to connect
0806 03:00:04.664249 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 03:00:04.664459 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:00:04.664879 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:00:04.664884 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:00:04.665331 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:00:04.665485 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:04.665494 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:04.666704 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:04.666709 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:04.667859 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:04.667872 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:04.669063 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:04.669072 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:04.669381 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:04.670586 (+ 1205us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:00:04.670645 (+ 59us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:00:04.676837 (+ 6192us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.74.254:45993: Server oryx/127.31.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1142,"thread_start_us":880,"threads_started":1}
W20260806 03:00:04.677439 9075 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:00:04.663127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.195:56485 (local address 127.31.74.254:45993)
0806 03:00:04.663873 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0806 03:00:04.663881 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 03:00:04.664378 (+ 497us) server_negotiation.cc:407] Connection header received
0806 03:00:04.664609 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:00:04.664615 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:00:04.664701 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:00:04.664821 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:00:04.665644 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:04.666479 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:04.668005 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:04.668818 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:04.668970 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:04.677152 (+ 8182us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.74.195:56485: BlockingRecv error: recv got EOF from 127.31.74.195:56485 (error 108)
Metrics: {"server-negotiator.queue_time_us":175}
Aug 06 03:00:04 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.193@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:00:04 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.193@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260806 03:00:04.782362 9097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:00:04.769316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:45993 (local address 127.31.74.193:55147)
0806 03:00:04.770153 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0806 03:00:04.770173 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 03:00:04.770309 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:00:04.770914 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:00:04.770924 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:00:04.771466 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:00:04.771665 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:04.771675 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:04.773178 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:04.773183 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:04.774386 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:04.774402 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:04.775536 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:04.775544 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:04.775750 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:04.776607 (+ 857us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:00:04.776625 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:00:04.782044 (+ 5419us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.74.254:45993: Server oryx/127.31.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":331,"threads_started":1}
W20260806 03:00:04.782886 9075 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:00:04.769428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.193:55147 (local address 127.31.74.254:45993)
0806 03:00:04.769662 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0806 03:00:04.769671 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 03:00:04.770286 (+ 615us) server_negotiation.cc:407] Connection header received
0806 03:00:04.770479 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:00:04.770484 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:00:04.770555 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:00:04.770764 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:00:04.771965 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:04.772977 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:04.774563 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:04.775317 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:04.775515 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:04.782687 (+ 7172us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.74.193:55147: BlockingRecv error: recv got EOF from 127.31.74.193:55147 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Aug 06 03:00:05 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.194@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:00:05 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.194@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260806 03:00:05.251511 9098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:00:05.235311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:45993 (local address 127.31.74.194:50741)
0806 03:00:05.236267 (+ 956us) negotiation.cc:107] Waiting for socket to connect
0806 03:00:05.236286 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 03:00:05.236424 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:00:05.236816 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:00:05.236820 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:00:05.237280 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:00:05.237498 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:05.237508 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:05.238830 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:05.238835 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:05.239906 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:05.239920 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:05.240778 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:05.240784 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:05.240917 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:05.245007 (+ 4090us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:00:05.245036 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:00:05.251089 (+ 6053us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.74.254:45993: Server oryx/127.31.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":825,"thread_start_us":387,"threads_started":1}
W20260806 03:00:05.251703 9075 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:00:05.235420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.194:50741 (local address 127.31.74.254:45993)
0806 03:00:05.235890 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0806 03:00:05.235898 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 03:00:05.236447 (+ 549us) server_negotiation.cc:407] Connection header received
0806 03:00:05.236535 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:00:05.236539 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:00:05.236603 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:00:05.236682 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:00:05.237686 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:05.238645 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:05.240085 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:05.240577 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:05.240696 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:05.251378 (+ 10682us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.74.194:50741: BlockingRecv error: recv got EOF from 127.31.74.194:50741 (error 108)
Metrics: {"server-negotiator.queue_time_us":315}
Aug 06 03:00:05 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.195@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:00:05 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.195@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260806 03:00:05.694461 9099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:00:05.678370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:45993 (local address 127.31.74.195:33641)
0806 03:00:05.679186 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0806 03:00:05.679204 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 03:00:05.679465 (+ 261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:00:05.680405 (+ 940us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:00:05.680414 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:00:05.681249 (+ 835us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:00:05.681556 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:05.681571 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:05.683874 (+ 2303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:05.683880 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:05.685582 (+ 1702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:05.685608 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:05.687800 (+ 2192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:05.687807 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:05.687955 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:05.688867 (+ 912us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:00:05.688888 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:00:05.694116 (+ 5228us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.74.254:45993: Server oryx/127.31.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":688,"thread_start_us":449,"threads_started":1}
W20260806 03:00:05.694859 9075 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:00:05.678439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.195:33641 (local address 127.31.74.254:45993)
0806 03:00:05.678693 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0806 03:00:05.678701 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 03:00:05.679318 (+ 617us) server_negotiation.cc:407] Connection header received
0806 03:00:05.679670 (+ 352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:00:05.679679 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:00:05.679797 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:00:05.679918 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:00:05.681889 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:05.683579 (+ 1690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:05.685906 (+ 2327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:05.687520 (+ 1614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:05.687720 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:05.694408 (+ 6688us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.74.195:33641: BlockingRecv error: recv got EOF from 127.31.74.195:33641 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 06 03:00:05 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.193@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:00:05 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.193@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260806 03:00:05.797293 9100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:00:05.783976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:45993 (local address 127.31.74.193:49647)
0806 03:00:05.784646 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0806 03:00:05.784672 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 03:00:05.785078 (+ 406us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:00:05.785585 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:00:05.785588 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:00:05.786037 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:00:05.786321 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:05.786354 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:05.788134 (+ 1780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:05.788138 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:05.789181 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:05.789193 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:05.790496 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:05.790501 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:05.790721 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:05.791605 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:00:05.791623 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:00:05.796942 (+ 5319us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.74.254:45993: Server oryx/127.31.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":390,"threads_started":1}
W20260806 03:00:05.799772 9075 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:00:05.784153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.193:49647 (local address 127.31.74.254:45993)
0806 03:00:05.784377 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0806 03:00:05.784386 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 03:00:05.784791 (+ 405us) server_negotiation.cc:407] Connection header received
0806 03:00:05.785323 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:00:05.785332 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:00:05.785430 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:00:05.785547 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:00:05.786528 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:05.787941 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:05.789373 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:05.790232 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:05.790395 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:05.799279 (+ 8884us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.74.193:49647: BlockingRecv error: recv got EOF from 127.31.74.193:49647 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 06 03:00:06 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.194@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:00:06 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.194@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260806 03:00:06.265306 9101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:00:06.252942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:45993 (local address 127.31.74.194:55433)
0806 03:00:06.253856 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0806 03:00:06.253876 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 03:00:06.254077 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:00:06.254515 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:00:06.254520 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:00:06.255054 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:00:06.255239 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:06.255248 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:06.256575 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:06.256582 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:06.257825 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:06.257840 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:06.258757 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:06.258766 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:06.258958 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:06.259765 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:00:06.259790 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:00:06.264695 (+ 4905us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.74.254:45993: Server oryx/127.31.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":707,"thread_start_us":384,"threads_started":1}
W20260806 03:00:06.265506 9075 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:00:06.253271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.194:55433 (local address 127.31.74.254:45993)
0806 03:00:06.253516 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0806 03:00:06.253526 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 03:00:06.254011 (+ 485us) server_negotiation.cc:407] Connection header received
0806 03:00:06.254233 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:00:06.254239 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:00:06.254356 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:00:06.254479 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:00:06.255423 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:06.256407 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:06.258018 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:06.258501 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:06.258681 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:06.265213 (+ 6532us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.74.194:55433: BlockingRecv error: recv got EOF from 127.31.74.194:55433 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Aug 06 03:00:06 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.195@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:00:06 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.195@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260806 03:00:06.709101 9102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:00:06.695719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:45993 (local address 127.31.74.195:44409)
0806 03:00:06.696361 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0806 03:00:06.696380 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 03:00:06.696566 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:00:06.697538 (+ 972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:00:06.697544 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:00:06.698077 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:00:06.698309 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:06.698321 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:06.700109 (+ 1788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:06.700142 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:06.701820 (+ 1678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:06.701838 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:06.702840 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:06.702846 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:06.702992 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:06.703795 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:00:06.703824 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:00:06.708748 (+ 4924us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.74.254:45993: Server oryx/127.31.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":319,"threads_started":1}
W20260806 03:00:06.709450 9075 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:00:06.695850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.195:44409 (local address 127.31.74.254:45993)
0806 03:00:06.696124 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0806 03:00:06.696132 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 03:00:06.696476 (+ 344us) server_negotiation.cc:407] Connection header received
0806 03:00:06.696888 (+ 412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:00:06.696899 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:00:06.697019 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:00:06.697327 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:00:06.698676 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:06.699846 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:06.702031 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:06.702562 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:06.702725 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:06.709096 (+ 6371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.74.195:44409: BlockingRecv error: recv got EOF from 127.31.74.195:44409 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 06 03:00:06 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.193@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:00:06 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.193@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260806 03:00:06.812601 9103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:00:06.799564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:45993 (local address 127.31.74.193:44229)
0806 03:00:06.800205 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0806 03:00:06.800222 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 03:00:06.800348 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:00:06.800969 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:00:06.800975 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:00:06.802120 (+ 1145us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:00:06.802402 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:06.802417 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:06.804565 (+ 2148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:06.804577 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:06.805668 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:06.805684 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:06.806543 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:06.806548 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:06.806780 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:06.807965 (+ 1185us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:00:06.807987 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:00:06.812319 (+ 4332us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.74.254:45993: Server oryx/127.31.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":377,"threads_started":1}
W20260806 03:00:06.813177 9075 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:00:06.799706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.193:44229 (local address 127.31.74.254:45993)
0806 03:00:06.799922 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0806 03:00:06.799927 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 03:00:06.800321 (+ 394us) server_negotiation.cc:407] Connection header received
0806 03:00:06.800516 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:00:06.800520 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:00:06.800592 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:00:06.800711 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:00:06.802675 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:06.804339 (+ 1664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:06.805850 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:06.806329 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:06.806454 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:06.812971 (+ 6517us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.74.193:44229: BlockingRecv error: recv got EOF from 127.31.74.193:44229 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 06 03:00:07 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.194@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:00:07 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.194@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260806 03:00:07.279186 9104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:00:07.267170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:45993 (local address 127.31.74.194:47469)
0806 03:00:07.267742 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0806 03:00:07.267760 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 03:00:07.267876 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:00:07.268349 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:00:07.268353 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:00:07.268776 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:00:07.268934 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:07.268943 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:07.270684 (+ 1741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:07.270690 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:07.272339 (+ 1649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:07.272359 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:07.273349 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:07.273355 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:07.273492 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:07.274292 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:00:07.274312 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:00:07.278756 (+ 4444us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.74.254:45993: Server oryx/127.31.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":303,"threads_started":1}
W20260806 03:00:07.279279 9075 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:00:07.267597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.194:47469 (local address 127.31.74.254:45993)
0806 03:00:07.267807 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0806 03:00:07.267811 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 03:00:07.267834 (+ 23us) server_negotiation.cc:407] Connection header received
0806 03:00:07.268031 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:00:07.268036 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:00:07.268112 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:00:07.268214 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:00:07.269135 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:07.270476 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:07.272527 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:07.273108 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:07.273229 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:07.279006 (+ 5777us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.74.194:47469: BlockingRecv error: recv got EOF from 127.31.74.194:47469 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 06 03:00:07 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.195@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:00:07 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.195@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
W20260806 03:00:07.723272 9075 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:00:07.710888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.195:36849 (local address 127.31.74.254:45993)
0806 03:00:07.711135 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0806 03:00:07.711141 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 03:00:07.711462 (+ 321us) server_negotiation.cc:407] Connection header received
0806 03:00:07.711568 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:00:07.711572 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:00:07.711635 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:00:07.711746 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:00:07.713419 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:07.714418 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:07.715967 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:07.716536 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:07.716685 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:07.722678 (+ 5993us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.74.195:36849: BlockingRecv error: recv got EOF from 127.31.74.195:36849 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 03:00:07.723908 9105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:00:07.710727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:45993 (local address 127.31.74.195:36849)
0806 03:00:07.711329 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0806 03:00:07.711346 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 03:00:07.711464 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:00:07.711963 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:00:07.711969 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:00:07.712954 (+ 985us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:00:07.713196 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:07.713216 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:07.714666 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:07.714672 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:07.715788 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:07.715803 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:07.716755 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:07.716760 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:07.716905 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:07.717615 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:00:07.717639 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:00:07.722390 (+ 4751us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.74.254:45993: Server oryx/127.31.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":325,"threads_started":1}
Aug 06 03:00:07 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.193@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:00:07 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.193@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
W20260806 03:00:07.829752 9075 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:00:07.814349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.193:54381 (local address 127.31.74.254:45993)
0806 03:00:07.814627 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0806 03:00:07.814637 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 03:00:07.815316 (+ 679us) server_negotiation.cc:407] Connection header received
0806 03:00:07.815550 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:00:07.815554 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:00:07.815622 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:00:07.815708 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:00:07.816831 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:07.817660 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:07.820006 (+ 2346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:07.820594 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:07.820759 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:07.829427 (+ 8668us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.74.193:54381: BlockingRecv error: recv got EOF from 127.31.74.193:54381 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 03:00:07.829748 9106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:00:07.814158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:45993 (local address 127.31.74.193:54381)
0806 03:00:07.815184 (+ 1026us) negotiation.cc:107] Waiting for socket to connect
0806 03:00:07.815231 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 03:00:07.815397 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:00:07.815797 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:00:07.815800 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:00:07.816254 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:00:07.816406 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:07.816415 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:07.817832 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:07.817836 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:07.819405 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:07.819442 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:07.820957 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:07.820963 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:07.821134 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:07.822075 (+ 941us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:00:07.822102 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:00:07.828686 (+ 6584us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.74.254:45993: Server oryx/127.31.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":908,"thread_start_us":791,"threads_started":1}
Aug 06 03:00:08 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.194@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:00:08 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.194@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260806 03:00:08.293208 9107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:00:08.280709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:45993 (local address 127.31.74.194:51889)
0806 03:00:08.281482 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0806 03:00:08.281513 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0806 03:00:08.281734 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:00:08.282154 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:00:08.282160 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:00:08.283093 (+ 933us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:00:08.283343 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:08.283359 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:08.284956 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:08.284966 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:08.286371 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:08.286389 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:08.287303 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:08.287310 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:08.287495 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:08.288182 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:00:08.288206 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:00:08.292931 (+ 4725us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.74.254:45993: Server oryx/127.31.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":642,"thread_start_us":453,"threads_started":1}
W20260806 03:00:08.293627 9075 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:00:08.280908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.194:51889 (local address 127.31.74.254:45993)
0806 03:00:08.281299 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0806 03:00:08.281309 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 03:00:08.281641 (+ 332us) server_negotiation.cc:407] Connection header received
0806 03:00:08.281896 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:00:08.281911 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:00:08.281982 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:00:08.282118 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:00:08.283654 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:08.284725 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:08.286606 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:08.287105 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:08.287269 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:08.293237 (+ 5968us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.74.194:51889: BlockingRecv error: recv got EOF from 127.31.74.194:51889 (error 108)
Metrics: {"server-negotiator.queue_time_us":127}
Aug 06 03:00:08 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.195@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:00:08 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.195@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260806 03:00:08.737982 9108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:00:08.725603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:45993 (local address 127.31.74.195:52601)
0806 03:00:08.726236 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0806 03:00:08.726263 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0806 03:00:08.726415 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:00:08.726837 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:00:08.726841 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:00:08.727553 (+ 712us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:00:08.727734 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:08.727743 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:08.729030 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:08.729035 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:08.730180 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:08.730193 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:08.731120 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:08.731125 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:08.731244 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:08.732230 (+ 986us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:00:08.732250 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:00:08.737654 (+ 5404us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.74.254:45993: Server oryx/127.31.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":369,"threads_started":1}
W20260806 03:00:08.738390 9075 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:00:08.725717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.195:52601 (local address 127.31.74.254:45993)
0806 03:00:08.726289 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0806 03:00:08.726295 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 03:00:08.726323 (+ 28us) server_negotiation.cc:407] Connection header received
0806 03:00:08.726545 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:00:08.726551 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:00:08.726661 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:00:08.726775 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:00:08.728037 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:08.728860 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:08.730369 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:08.730945 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:08.731054 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:08.737996 (+ 6942us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.74.195:52601: BlockingRecv error: recv got EOF from 127.31.74.195:52601 (error 108)
Metrics: {"server-negotiator.queue_time_us":397}
Aug 06 03:00:08 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.193@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:00:08 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.193@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260806 03:00:08.844075 9109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:00:08.831425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:45993 (local address 127.31.74.193:44099)
0806 03:00:08.832281 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0806 03:00:08.832328 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 03:00:08.832500 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:00:08.832964 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:00:08.832972 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:00:08.833657 (+ 685us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:00:08.833882 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:08.833893 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:08.835157 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:08.835161 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:08.836094 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:08.836106 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:08.837021 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:08.837027 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:08.837170 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:08.837962 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:00:08.837982 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:00:08.843546 (+ 5564us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.74.254:45993: Server oryx/127.31.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":717,"thread_start_us":385,"threads_started":1}
W20260806 03:00:08.844156 9075 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:00:08.831702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.193:44099 (local address 127.31.74.254:45993)
0806 03:00:08.831929 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0806 03:00:08.831935 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 03:00:08.832452 (+ 517us) server_negotiation.cc:407] Connection header received
0806 03:00:08.832632 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:00:08.832638 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:00:08.832781 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:00:08.832885 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:00:08.834051 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:08.835003 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:08.836266 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:08.836745 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:08.836906 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:08.843908 (+ 7002us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.74.193:44099: BlockingRecv error: recv got EOF from 127.31.74.193:44099 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Aug 06 03:00:09 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.194@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:00:09 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.194@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260806 03:00:09.305814 9110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:00:09.294351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:45993 (local address 127.31.74.194:46837)
0806 03:00:09.295080 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0806 03:00:09.295101 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 03:00:09.295227 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:00:09.295565 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:00:09.295569 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:00:09.296013 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:00:09.296200 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:09.296210 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:09.297448 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:09.297453 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:09.298528 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:09.298541 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:09.299498 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:09.299504 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:09.299693 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:09.300513 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:00:09.300534 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:00:09.305536 (+ 5002us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.74.254:45993: Server oryx/127.31.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":456,"threads_started":1}
W20260806 03:00:09.306066 9075 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:00:09.294446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.194:46837 (local address 127.31.74.254:45993)
0806 03:00:09.294803 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0806 03:00:09.294814 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 03:00:09.295214 (+ 400us) server_negotiation.cc:407] Connection header received
0806 03:00:09.295351 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:00:09.295356 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:00:09.295426 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:00:09.295523 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:00:09.296412 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:09.297279 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:09.298823 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:09.299303 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:09.299424 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:09.305781 (+ 6357us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.74.194:46837: BlockingRecv error: recv got EOF from 127.31.74.194:46837 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
Aug 06 03:00:09 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.195@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:00:09 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.195@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260806 03:00:09.750556 9111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:00:09.739310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:45993 (local address 127.31.74.195:58139)
0806 03:00:09.740036 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0806 03:00:09.740054 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 03:00:09.740271 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:00:09.740660 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:00:09.740665 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:00:09.741168 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:00:09.741362 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:09.741371 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:09.742777 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:09.742788 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:09.744142 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:09.744154 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:09.744934 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:09.744939 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:09.745063 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:09.745887 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:00:09.745906 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:00:09.750304 (+ 4398us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.74.254:45993: Server oryx/127.31.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":485,"threads_started":1}
W20260806 03:00:09.750849 9075 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:00:09.739524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.195:58139 (local address 127.31.74.254:45993)
0806 03:00:09.739802 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0806 03:00:09.739812 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 03:00:09.740162 (+ 350us) server_negotiation.cc:407] Connection header received
0806 03:00:09.740432 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:00:09.740437 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:00:09.740505 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:00:09.740626 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:00:09.741530 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:09.742525 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:09.744305 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:09.744754 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:09.744855 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:09.750533 (+ 5678us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.74.195:58139: BlockingRecv error: recv got EOF from 127.31.74.195:58139 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
Aug 06 03:00:09 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.193@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:00:09 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.193@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260806 03:00:09.859808 9112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:00:09.845664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:45993 (local address 127.31.74.193:57793)
0806 03:00:09.846272 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0806 03:00:09.846289 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 03:00:09.846434 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:00:09.847287 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:00:09.847291 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:00:09.847907 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:00:09.848161 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:09.848176 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:09.849915 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:09.849923 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:09.851756 (+ 1833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:09.851775 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:09.852623 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:09.852628 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:09.852796 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:09.853576 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:00:09.853595 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:00:09.859348 (+ 5753us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.74.254:45993: Server oryx/127.31.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":395,"threads_started":1}
W20260806 03:00:09.860031 9075 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:00:09.845773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.193:57793 (local address 127.31.74.254:45993)
0806 03:00:09.846081 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0806 03:00:09.846089 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 03:00:09.846524 (+ 435us) server_negotiation.cc:407] Connection header received
0806 03:00:09.846644 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:00:09.846648 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:00:09.846722 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:00:09.846832 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:00:09.848638 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:09.849696 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:09.851942 (+ 2246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:09.852454 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:09.852564 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:09.859621 (+ 7057us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.74.193:57793: BlockingRecv error: recv got EOF from 127.31.74.193:57793 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Aug 06 03:00:10 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.194@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:00:10 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.194@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260806 03:00:10.323312 9113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:00:10.307017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:45993 (local address 127.31.74.194:38917)
0806 03:00:10.307633 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0806 03:00:10.307652 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 03:00:10.307784 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:00:10.308157 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:00:10.308161 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:00:10.308581 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:00:10.308811 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:10.308822 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:10.310063 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:10.310068 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:10.311226 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:10.311240 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:10.312081 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:10.312090 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:10.312307 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:10.318432 (+ 6125us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:00:10.318454 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:00:10.322993 (+ 4539us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.74.254:45993: Server oryx/127.31.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":365,"threads_started":1}
W20260806 03:00:10.323619 9075 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:00:10.307141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.194:38917 (local address 127.31.74.254:45993)
0806 03:00:10.307334 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0806 03:00:10.307340 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 03:00:10.307780 (+ 440us) server_negotiation.cc:407] Connection header received
0806 03:00:10.307891 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:00:10.307896 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:00:10.307967 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:00:10.308068 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:00:10.308984 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:10.309891 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:10.311444 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:10.311883 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:10.311991 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:10.323242 (+ 11251us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.74.194:38917: BlockingRecv error: recv got EOF from 127.31.74.194:38917 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 06 03:00:10 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.195@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:00:10 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.195@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260806 03:00:10.764909 9114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:00:10.752074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:45993 (local address 127.31.74.195:60823)
0806 03:00:10.752733 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0806 03:00:10.752796 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0806 03:00:10.752938 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:00:10.753689 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:00:10.753698 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:00:10.754748 (+ 1050us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:00:10.755027 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:10.755043 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:10.756243 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:10.756250 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:10.757245 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:10.757258 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:10.758376 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:10.758383 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:10.758623 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:10.759674 (+ 1051us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:00:10.759710 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:00:10.764605 (+ 4895us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.74.254:45993: Server oryx/127.31.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":398,"threads_started":1}
W20260806 03:00:10.765931 9075 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:00:10.752275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.195:60823 (local address 127.31.74.254:45993)
0806 03:00:10.752504 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0806 03:00:10.752509 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 03:00:10.752925 (+ 416us) server_negotiation.cc:407] Connection header received
0806 03:00:10.753043 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:00:10.753048 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:00:10.753323 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:00:10.753615 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:00:10.755202 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:10.756005 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:10.757406 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:10.758190 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:10.758358 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:10.765239 (+ 6881us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.74.195:60823: BlockingRecv error: recv got EOF from 127.31.74.195:60823 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 06 03:00:10 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.193@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:00:10 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.193@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260806 03:00:10.872937 9115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:00:10.861763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:45993 (local address 127.31.74.193:55125)
0806 03:00:10.862498 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0806 03:00:10.862520 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 03:00:10.862742 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:00:10.863110 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:00:10.863115 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:00:10.863626 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:00:10.863802 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:10.863811 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:10.864970 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:10.864977 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:10.866056 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:10.866070 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:10.867004 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:10.867010 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:10.867213 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:10.868065 (+ 852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:00:10.868084 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:00:10.872583 (+ 4499us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.74.254:45993: Server oryx/127.31.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":384,"threads_started":1}
W20260806 03:00:10.873544 9075 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:00:10.861916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.193:55125 (local address 127.31.74.254:45993)
0806 03:00:10.862301 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0806 03:00:10.862309 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 03:00:10.862644 (+ 335us) server_negotiation.cc:407] Connection header received
0806 03:00:10.862884 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:00:10.862895 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:00:10.862965 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:00:10.863078 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:00:10.863965 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:10.864813 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:10.866235 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:10.866826 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:10.866970 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:10.873267 (+ 6297us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.74.193:55125: BlockingRecv error: recv got EOF from 127.31.74.193:55125 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
Aug 06 03:00:11 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.194@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:00:11 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.194@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260806 03:00:11.336423 9116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:00:11.324827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:45993 (local address 127.31.74.194:43733)
0806 03:00:11.325612 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0806 03:00:11.325635 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 03:00:11.325816 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:00:11.326309 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:00:11.326314 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:00:11.326917 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:00:11.327086 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:11.327096 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:11.328355 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:11.328360 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:11.329436 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:11.329450 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:11.330394 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:11.330401 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:11.330653 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:11.331322 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:00:11.331340 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:00:11.335963 (+ 4623us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.74.254:45993: Server oryx/127.31.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":424,"threads_started":1}
W20260806 03:00:11.336491 9075 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:00:11.325103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.194:43733 (local address 127.31.74.254:45993)
0806 03:00:11.325475 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0806 03:00:11.325485 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 03:00:11.325807 (+ 322us) server_negotiation.cc:407] Connection header received
0806 03:00:11.325972 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:00:11.325977 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:00:11.326063 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:00:11.326185 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:00:11.327296 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:11.328185 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:11.329617 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:11.330213 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:11.330377 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:11.336243 (+ 5866us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.74.194:43733: BlockingRecv error: recv got EOF from 127.31.74.194:43733 (error 108)
Metrics: {"server-negotiator.queue_time_us":142}
Aug 06 03:00:11 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.195@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:00:11 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.195@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260806 03:00:11.777746 9117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:00:11.767158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:45993 (local address 127.31.74.195:43691)
0806 03:00:11.767722 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0806 03:00:11.767740 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 03:00:11.767875 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:00:11.768203 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:00:11.768206 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:00:11.768632 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:00:11.768777 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:11.768785 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:11.770428 (+ 1643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:11.770434 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:11.771758 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:11.771771 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:11.772562 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:11.772567 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:11.772743 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:11.773536 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:00:11.773559 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:00:11.777476 (+ 3917us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.74.254:45993: Server oryx/127.31.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":304,"threads_started":1}
W20260806 03:00:11.778120 9075 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:00:11.767384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.195:43691 (local address 127.31.74.254:45993)
0806 03:00:11.767670 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0806 03:00:11.767676 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 03:00:11.767891 (+ 215us) server_negotiation.cc:407] Connection header received
0806 03:00:11.767998 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:00:11.768003 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:00:11.768076 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:00:11.768161 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:00:11.769077 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:11.770227 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:11.771887 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:11.772379 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:11.772496 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:11.777761 (+ 5265us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.74.195:43691: BlockingRecv error: recv got EOF from 127.31.74.195:43691 (error 108)
Metrics: {"server-negotiator.queue_time_us":119}
Aug 06 03:00:11 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.193@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
Aug 06 03:00:11 dist-test-slave-dhph krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.74.193@KRBTEST.COM for oryx/127.31.74.254@KRBTEST.COM, Server not found in Kerberos database
I20260806 03:00:11.887331 9118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:00:11.874968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:45993 (local address 127.31.74.193:42295)
0806 03:00:11.875772 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0806 03:00:11.875801 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0806 03:00:11.875946 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:00:11.876985 (+ 1039us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:00:11.876990 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:00:11.877457 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:00:11.877651 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:11.877661 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:11.879209 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:11.879230 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:11.880288 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:11.880303 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:11.881229 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:11.881235 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:11.881458 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:11.882280 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:00:11.882301 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:00:11.886925 (+ 4624us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.74.254:45993: Server oryx/127.31.74.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":421,"threads_started":1}
W20260806 03:00:11.887914 9075 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:00:11.875114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.193:42295 (local address 127.31.74.254:45993)
0806 03:00:11.875464 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0806 03:00:11.875470 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 03:00:11.875935 (+ 465us) server_negotiation.cc:407] Connection header received
0806 03:00:11.876202 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:00:11.876209 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:00:11.876543 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:00:11.876849 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:00:11.877872 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:11.878996 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:11.880458 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:11.881033 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:11.881196 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:11.887197 (+ 6001us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.74.193:42295: BlockingRecv error: recv got EOF from 127.31.74.193:42295 (error 108)
Metrics: {"server-negotiator.queue_time_us":191}
I20260806 03:00:12.134001 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 8647
I20260806 03:00:12.152283 9120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:00:12.135843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40412 (local address 127.31.74.193:38781)
0806 03:00:12.136648 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0806 03:00:12.136656 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 03:00:12.136689 (+ 33us) server_negotiation.cc:407] Connection header received
0806 03:00:12.136761 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:00:12.136767 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:00:12.137024 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:00:12.137235 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:00:12.139321 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:12.141029 (+ 1708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:12.143106 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:12.143805 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:12.143952 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:12.147241 (+ 3289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:00:12.147252 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:00:12.147268 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:00:12.147777 (+ 509us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:00:12.151632 (+ 3855us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.31.74.193@KRBTEST.COM not found in keytab (ticket kvno 2)
0806 03:00:12.152007 (+ 375us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40412: Request ticket server kudu/127.31.74.193@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":550,"threads_started":1}
W20260806 03:00:12.152549 32043 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 8647: Not authorized: Client connection negotiation failed: client connection to 127.31.74.193:38781: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.31.74.193@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260806 03:00:12.152827 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 8647
W20260806 03:00:12.152422 9120 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40412: Request ticket server kudu/127.31.74.193@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260806 03:00:12.179952 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 8800
I20260806 03:00:12.199048 9121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:00:12.181179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47364 (local address 127.31.74.194:36887)
0806 03:00:12.182094 (+ 915us) server_negotiation.cc:206] Beginning negotiation
0806 03:00:12.182104 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 03:00:12.182137 (+ 33us) server_negotiation.cc:407] Connection header received
0806 03:00:12.182245 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:00:12.182255 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:00:12.182606 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:00:12.182844 (+ 238us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:00:12.184287 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:12.185853 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:12.189466 (+ 3613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:12.190304 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:12.190469 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:12.194335 (+ 3866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:00:12.194348 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:00:12.194365 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:00:12.194415 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:00:12.198284 (+ 3869us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.31.74.194@KRBTEST.COM not found in keytab (ticket kvno 2)
0806 03:00:12.198776 (+ 492us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47364: Request ticket server kudu/127.31.74.194@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":705,"thread_start_us":558,"threads_started":1}
W20260806 03:00:12.199183 9121 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47364: Request ticket server kudu/127.31.74.194@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260806 03:00:12.199836 32043 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 8800: Not authorized: Client connection negotiation failed: client connection to 127.31.74.194:36887: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.31.74.194@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260806 03:00:12.200044 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 8800
I20260806 03:00:12.224980 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 8937
I20260806 03:00:12.241042 9122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:00:12.226414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36290 (local address 127.31.74.195:39173)
0806 03:00:12.227272 (+ 858us) server_negotiation.cc:206] Beginning negotiation
0806 03:00:12.227280 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 03:00:12.227312 (+ 32us) server_negotiation.cc:407] Connection header received
0806 03:00:12.227384 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:00:12.227391 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:00:12.227754 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:00:12.228072 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:00:12.229694 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:12.231378 (+ 1684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:12.232849 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:12.233515 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:12.233674 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:12.236814 (+ 3140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:00:12.236824 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:00:12.236840 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:00:12.236896 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:00:12.240163 (+ 3267us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.31.74.195@KRBTEST.COM not found in keytab (ticket kvno 2)
0806 03:00:12.240588 (+ 425us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36290: Request ticket server kudu/127.31.74.195@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":483,"threads_started":1}
W20260806 03:00:12.241042 32043 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 8937: Not authorized: Client connection negotiation failed: client connection to 127.31.74.195:39173: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.31.74.195@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260806 03:00:12.241173 9122 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36290: Request ticket server kudu/127.31.74.195@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260806 03:00:12.241248 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 8937
I20260806 03:00:12.265564 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 8575
I20260806 03:00:12.267215 8639 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 03:00:12.398530 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 8575
2026-08-06T03:00:12Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (22864 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-06T03:00:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:00:12Z Disabled control of system clock
I20260806 03:00:12.512110 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785985105255475-32043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785985105255475-32043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785985105255475-32043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:37151
--webserver_interface=127.31.74.254
--webserver_port=0
--builtin_ntp_servers=127.31.74.212:43383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.74.254:37151
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260806 03:00:12.959550 9128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:00:12.959906 9128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:00:12.960014 9128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:00:12.969708 9128 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:00:12.970475 9128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:00:12.970539 9128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:00:12.970666 9128 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:00:12.970726 9128 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:00:12.981140 9128 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:43383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785985105255475-32043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785985105255475-32043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.74.254:37151
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:37151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.74.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785985105255475-32043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:00:12.982508 9128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:00:12.984465 9128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:00:13.001727 9133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:00:13.001950 9136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:00:13.002233 9134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:00:13.003618 9128 server_base.cc:1034] running on GCE node
I20260806 03:00:13.004865 9128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:00:13.006917 9128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:00:13.008199 9128 hybrid_clock.cc:648] HybridClock initialized: now 1785985213008157 us; error 51 us; skew 500 ppm
I20260806 03:00:13.008905 9128 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 03:00:13.012180 9128 webserver.cc:460] Webserver started at http://127.31.74.254:44235/ using document root <none> and password file <none>
I20260806 03:00:13.013567 9128 fs_manager.cc:359] Metadata directory not provided
I20260806 03:00:13.013773 9128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:00:13.014235 9128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:00:13.020352 9128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785985105255475-32043-0/minicluster-data/master-0/data/instance:
uuid: "6a9064d014dd4b559afdd51e36fdd039"
format_stamp: "Formatted at 2026-08-06 03:00:13 on dist-test-slave-dhph"
server_key: "1f6d024eb0704478124fba7fb346677c"
server_key_iv: "ca0440de39b4f2fcd399d536ca791f51"
server_key_version: "encryptionkey@0"
I20260806 03:00:13.021678 9128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785985105255475-32043-0/minicluster-data/master-0/wal/instance:
uuid: "6a9064d014dd4b559afdd51e36fdd039"
format_stamp: "Formatted at 2026-08-06 03:00:13 on dist-test-slave-dhph"
server_key: "1f6d024eb0704478124fba7fb346677c"
server_key_iv: "ca0440de39b4f2fcd399d536ca791f51"
server_key_version: "encryptionkey@0"
I20260806 03:00:13.028159 9128 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260806 03:00:13.032795 9143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:00:13.034970 9128 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260806 03:00:13.035193 9128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785985105255475-32043-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785985105255475-32043-0/minicluster-data/master-0/wal
uuid: "6a9064d014dd4b559afdd51e36fdd039"
format_stamp: "Formatted at 2026-08-06 03:00:13 on dist-test-slave-dhph"
server_key: "1f6d024eb0704478124fba7fb346677c"
server_key_iv: "ca0440de39b4f2fcd399d536ca791f51"
server_key_version: "encryptionkey@0"
I20260806 03:00:13.035452 9128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785985105255475-32043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785985105255475-32043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785985105255475-32043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:00:13.051478 9128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:00:13.052754 9128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:00:13.053114 9128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:00:13.076924 9128 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.254:37151
I20260806 03:00:13.076946 9194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.254:37151 every 8 connection(s)
I20260806 03:00:13.078534 9128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
I20260806 03:00:13.083053 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 9128
I20260806 03:00:13.083294 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785985105255475-32043-0/minicluster-data/master-0/wal/instance
I20260806 03:00:13.083946 9195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:00:13.084061 32043 external_mini_cluster.cc:1428] Setting key 354728649a5a6e5238659055996c4d56
I20260806 03:00:13.102351 9195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a9064d014dd4b559afdd51e36fdd039: Bootstrap starting.
I20260806 03:00:13.105872 9199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:00:13.087422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59908 (local address 127.31.74.254:37151)
0806 03:00:13.088833 (+ 1411us) server_negotiation.cc:206] Beginning negotiation
0806 03:00:13.088848 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 03:00:13.088922 (+ 74us) server_negotiation.cc:407] Connection header received
0806 03:00:13.092612 (+ 3690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:00:13.092645 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:00:13.093345 (+ 700us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:00:13.093763 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:00:13.095251 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:13.096834 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:13.099193 (+ 2359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:13.100313 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:13.100488 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:13.102261 (+ 1773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:00:13.102286 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:00:13.102314 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:00:13.103546 (+ 1232us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:00:13.103689 (+ 143us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:00:13.103731 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:00:13.103897 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0806 03:00:13.104297 (+ 400us) server_negotiation.cc:299] Negotiation successful
0806 03:00:13.104760 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":864,"thread_start_us":577,"threads_started":1}
I20260806 03:00:13.106428 9195 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a9064d014dd4b559afdd51e36fdd039: Neither blocks nor log segments found. Creating new log.
I20260806 03:00:13.108201 9195 log.cc:824] T 00000000000000000000000000000000 P 6a9064d014dd4b559afdd51e36fdd039: Log is configured to *not* fsync() on all Append() calls
I20260806 03:00:13.112207 9195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a9064d014dd4b559afdd51e36fdd039: No bootstrap required, opened a new log
I20260806 03:00:13.122668 9195 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a9064d014dd4b559afdd51e36fdd039 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a9064d014dd4b559afdd51e36fdd039" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 37151 } }
I20260806 03:00:13.123697 9195 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a9064d014dd4b559afdd51e36fdd039 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:00:13.123871 9195 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a9064d014dd4b559afdd51e36fdd039 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a9064d014dd4b559afdd51e36fdd039, State: Initialized, Role: FOLLOWER
I20260806 03:00:13.125427 9195 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a9064d014dd4b559afdd51e36fdd039 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a9064d014dd4b559afdd51e36fdd039" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 37151 } }
I20260806 03:00:13.125829 9195 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a9064d014dd4b559afdd51e36fdd039 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:00:13.126029 9195 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a9064d014dd4b559afdd51e36fdd039 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:00:13.126277 9195 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a9064d014dd4b559afdd51e36fdd039 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:00:13.129639 9195 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a9064d014dd4b559afdd51e36fdd039 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a9064d014dd4b559afdd51e36fdd039" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 37151 } }
I20260806 03:00:13.130452 9195 leader_election.cc:302] T 00000000000000000000000000000000 P 6a9064d014dd4b559afdd51e36fdd039 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a9064d014dd4b559afdd51e36fdd039; no voters:
I20260806 03:00:13.131220 9195 leader_election.cc:288] T 00000000000000000000000000000000 P 6a9064d014dd4b559afdd51e36fdd039 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:00:13.131496 9201 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a9064d014dd4b559afdd51e36fdd039 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:00:13.132380 9201 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a9064d014dd4b559afdd51e36fdd039 [term 1 LEADER]: Becoming Leader. State: Replica: 6a9064d014dd4b559afdd51e36fdd039, State: Running, Role: LEADER
I20260806 03:00:13.133078 9201 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a9064d014dd4b559afdd51e36fdd039 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a9064d014dd4b559afdd51e36fdd039" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 37151 } }
I20260806 03:00:13.133585 9195 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a9064d014dd4b559afdd51e36fdd039 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:00:13.137234 9202 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a9064d014dd4b559afdd51e36fdd039 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a9064d014dd4b559afdd51e36fdd039" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a9064d014dd4b559afdd51e36fdd039" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 37151 } } }
I20260806 03:00:13.137116 9203 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a9064d014dd4b559afdd51e36fdd039 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a9064d014dd4b559afdd51e36fdd039. Latest consensus state: current_term: 1 leader_uuid: "6a9064d014dd4b559afdd51e36fdd039" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a9064d014dd4b559afdd51e36fdd039" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 37151 } } }
I20260806 03:00:13.137693 9202 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a9064d014dd4b559afdd51e36fdd039 [sys.catalog]: This master's current role is: LEADER
I20260806 03:00:13.137719 9203 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a9064d014dd4b559afdd51e36fdd039 [sys.catalog]: This master's current role is: LEADER
I20260806 03:00:13.141191 9209 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:00:13.146864 9209 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:00:13.159533 9209 catalog_manager.cc:1347] Generated new cluster ID: fb7e90abe38a475ea0889551dfb4b224
I20260806 03:00:13.159782 9209 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:00:13.179859 9209 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:00:13.181401 9209 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:00:13.193616 9209 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a9064d014dd4b559afdd51e36fdd039: Generated new TSK 0
I20260806 03:00:13.194640 9209 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 03:00:13.205631 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.193:0
--local_ip_for_outbound_sockets=127.31.74.193
--webserver_interface=127.31.74.193
--webserver_port=0
--tserver_master_addrs=127.31.74.254:37151
--builtin_ntp_servers=127.31.74.212:43383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260806 03:00:13.593544 9220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:00:13.593859 9220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:00:13.593950 9220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:00:13.605729 9220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:00:13.605880 9220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:00:13.606003 9220 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.193
I20260806 03:00:13.620258 9220 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:43383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.74.193
--webserver_port=0
--tserver_master_addrs=127.31.74.254:37151
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.193
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:00:13.621917 9220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:00:13.623953 9220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:00:13.640226 9225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:00:13.640268 9226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:00:13.640226 9228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:00:13.642004 9220 server_base.cc:1034] running on GCE node
I20260806 03:00:13.643128 9220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:00:13.644346 9220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:00:13.645741 9220 hybrid_clock.cc:648] HybridClock initialized: now 1785985213645667 us; error 41 us; skew 500 ppm
I20260806 03:00:13.646303 9220 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 03:00:13.649174 9220 webserver.cc:460] Webserver started at http://127.31.74.193:43155/ using document root <none> and password file <none>
I20260806 03:00:13.650243 9220 fs_manager.cc:359] Metadata directory not provided
I20260806 03:00:13.650434 9220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:00:13.650969 9220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:00:13.655663 9220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-0/data/instance:
uuid: "1b15b6ba05ed47348055b094eb2c0e81"
format_stamp: "Formatted at 2026-08-06 03:00:13 on dist-test-slave-dhph"
server_key: "6e915b04fd0ce2cf772567a2bbbca426"
server_key_iv: "d594a8da07693cfba04ad861cf0765bf"
server_key_version: "encryptionkey@0"
I20260806 03:00:13.656754 9220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-0/wal/instance:
uuid: "1b15b6ba05ed47348055b094eb2c0e81"
format_stamp: "Formatted at 2026-08-06 03:00:13 on dist-test-slave-dhph"
server_key: "6e915b04fd0ce2cf772567a2bbbca426"
server_key_iv: "d594a8da07693cfba04ad861cf0765bf"
server_key_version: "encryptionkey@0"
I20260806 03:00:13.664009 9220 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260806 03:00:13.668138 9234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:00:13.670820 9220 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260806 03:00:13.671054 9220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-0/wal
uuid: "1b15b6ba05ed47348055b094eb2c0e81"
format_stamp: "Formatted at 2026-08-06 03:00:13 on dist-test-slave-dhph"
server_key: "6e915b04fd0ce2cf772567a2bbbca426"
server_key_iv: "d594a8da07693cfba04ad861cf0765bf"
server_key_version: "encryptionkey@0"
I20260806 03:00:13.671337 9220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:00:13.700352 9220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:00:13.701349 9220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:00:13.701694 9220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:00:13.702961 9220 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:00:13.704703 9220 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:00:13.704836 9220 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:00:13.704993 9220 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:00:13.705089 9220 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:00:13.744815 9220 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.193:45433
I20260806 03:00:13.744920 9348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.193:45433 every 8 connection(s)
I20260806 03:00:13.746176 9220 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
I20260806 03:00:13.748780 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 9220
I20260806 03:00:13.749009 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-0/wal/instance
I20260806 03:00:13.749877 32043 external_mini_cluster.cc:1428] Setting key 44bb712ed726c8e55d0f4d8891968e0c
I20260806 03:00:13.752774 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.194:0
--local_ip_for_outbound_sockets=127.31.74.194
--webserver_interface=127.31.74.194
--webserver_port=0
--tserver_master_addrs=127.31.74.254:37151
--builtin_ntp_servers=127.31.74.212:43383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260806 03:00:13.763695 9351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:00:13.749284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.193:53479 (local address 127.31.74.254:37151)
0806 03:00:13.749878 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0806 03:00:13.749885 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 03:00:13.751538 (+ 1653us) server_negotiation.cc:407] Connection header received
0806 03:00:13.753837 (+ 2299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:00:13.753843 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:00:13.753945 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:00:13.754065 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:00:13.756511 (+ 2446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:13.757247 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:13.759807 (+ 2560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:13.760418 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:13.760602 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:13.762701 (+ 2099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:00:13.762707 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:00:13.762710 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:00:13.762736 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:00:13.762798 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:00:13.762813 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:00:13.762927 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0806 03:00:13.763190 (+ 263us) server_negotiation.cc:299] Negotiation successful
0806 03:00:13.763330 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":288,"threads_started":1}
I20260806 03:00:13.764950 9352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:00:13.749974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:37151 (local address 127.31.74.193:53479)
0806 03:00:13.751348 (+ 1374us) negotiation.cc:107] Waiting for socket to connect
0806 03:00:13.751402 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 03:00:13.753533 (+ 2131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:00:13.755046 (+ 1513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:00:13.755063 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:00:13.755238 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:00:13.756280 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:13.756305 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:13.757476 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:13.757480 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:13.759233 (+ 1753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:13.759266 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:13.760799 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:13.760817 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:13.761168 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:13.762141 (+ 973us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 03:00:13.762145 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:00:13.762202 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 03:00:13.762316 (+ 114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:00:13.762932 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:00:13.762947 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:00:13.762957 (+ 10us) client_negotiation.cc:769] Sending connection context
0806 03:00:13.763163 (+ 206us) client_negotiation.cc:240] Negotiation successful
0806 03:00:13.763706 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":752,"thread_start_us":459,"threads_started":1}
I20260806 03:00:13.767151 9349 heartbeater.cc:347] Connected to a master server at 127.31.74.254:37151
I20260806 03:00:13.767509 9349 heartbeater.cc:464] Registering TS with master...
I20260806 03:00:13.769196 9349 heartbeater.cc:511] Master 127.31.74.254:37151 requested a full tablet report, sending...
I20260806 03:00:13.772655 9160 ts_manager.cc:194] Registered new tserver with Master: 1b15b6ba05ed47348055b094eb2c0e81 (127.31.74.193:45433)
I20260806 03:00:13.775976 9160 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.74.193:53479
W20260806 03:00:14.159668 9353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:00:14.160013 9353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:00:14.160118 9353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:00:14.169236 9353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:00:14.169382 9353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:00:14.169504 9353 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.194
I20260806 03:00:14.184594 9353 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:43383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.74.194
--webserver_port=0
--tserver_master_addrs=127.31.74.254:37151
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.194
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:00:14.185957 9353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:00:14.188066 9353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:00:14.205914 9358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:00:14.206656 9359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:00:14.206768 9361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:00:14.208195 9353 server_base.cc:1034] running on GCE node
I20260806 03:00:14.209429 9353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:00:14.210807 9353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:00:14.212093 9353 hybrid_clock.cc:648] HybridClock initialized: now 1785985214212020 us; error 62 us; skew 500 ppm
I20260806 03:00:14.212571 9353 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 03:00:14.215628 9353 webserver.cc:460] Webserver started at http://127.31.74.194:43589/ using document root <none> and password file <none>
I20260806 03:00:14.216496 9353 fs_manager.cc:359] Metadata directory not provided
I20260806 03:00:14.216648 9353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:00:14.217025 9353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:00:14.220417 9353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-1/data/instance:
uuid: "6dbad88aa22c4681940263ccbcbb0edf"
format_stamp: "Formatted at 2026-08-06 03:00:14 on dist-test-slave-dhph"
server_key: "4d8664bbae36ea1491c04d9d87c2ee11"
server_key_iv: "35d9131709739b8702edeec83b43cf28"
server_key_version: "encryptionkey@0"
I20260806 03:00:14.221369 9353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-1/wal/instance:
uuid: "6dbad88aa22c4681940263ccbcbb0edf"
format_stamp: "Formatted at 2026-08-06 03:00:14 on dist-test-slave-dhph"
server_key: "4d8664bbae36ea1491c04d9d87c2ee11"
server_key_iv: "35d9131709739b8702edeec83b43cf28"
server_key_version: "encryptionkey@0"
I20260806 03:00:14.228730 9353 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260806 03:00:14.235232 9367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:00:14.237195 9353 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.000s sys 0.004s
I20260806 03:00:14.237470 9353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-1/wal
uuid: "6dbad88aa22c4681940263ccbcbb0edf"
format_stamp: "Formatted at 2026-08-06 03:00:14 on dist-test-slave-dhph"
server_key: "4d8664bbae36ea1491c04d9d87c2ee11"
server_key_iv: "35d9131709739b8702edeec83b43cf28"
server_key_version: "encryptionkey@0"
I20260806 03:00:14.237753 9353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:00:14.257697 9353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:00:14.258962 9353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:00:14.259305 9353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:00:14.260391 9353 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:00:14.261979 9353 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:00:14.262070 9353 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:00:14.262215 9353 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:00:14.262290 9353 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:00:14.306905 9353 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.194:40925
I20260806 03:00:14.306972 9481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.194:40925 every 8 connection(s)
I20260806 03:00:14.308391 9353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
I20260806 03:00:14.316023 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 9353
I20260806 03:00:14.316184 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-1/wal/instance
I20260806 03:00:14.316579 32043 external_mini_cluster.cc:1428] Setting key 67ac4e91841cc03ebbea67b7ade8c43b
I20260806 03:00:14.319741 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.195:0
--local_ip_for_outbound_sockets=127.31.74.195
--webserver_interface=127.31.74.195
--webserver_port=0
--tserver_master_addrs=127.31.74.254:37151
--builtin_ntp_servers=127.31.74.212:43383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260806 03:00:14.324627 9484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:00:14.312071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.194:50115 (local address 127.31.74.254:37151)
0806 03:00:14.312569 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0806 03:00:14.312573 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 03:00:14.313430 (+ 857us) server_negotiation.cc:407] Connection header received
0806 03:00:14.315382 (+ 1952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:00:14.315387 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:00:14.315488 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:00:14.315610 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:00:14.317571 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:14.318429 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:14.320534 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:14.321452 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:14.321640 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:14.323480 (+ 1840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:00:14.323489 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:00:14.323497 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:00:14.323528 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:00:14.323599 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:00:14.323620 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:00:14.323724 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 03:00:14.323981 (+ 257us) server_negotiation.cc:299] Negotiation successful
0806 03:00:14.324109 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":276,"threads_started":1}
I20260806 03:00:14.325472 9485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:00:14.312358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:37151 (local address 127.31.74.194:50115)
0806 03:00:14.313281 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0806 03:00:14.313317 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0806 03:00:14.315094 (+ 1777us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:00:14.316518 (+ 1424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:00:14.316527 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:00:14.316632 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:00:14.317407 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:14.317429 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:14.318654 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:14.318662 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:14.320360 (+ 1698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:14.320376 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:14.321740 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:14.321745 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:14.322101 (+ 356us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:14.323083 (+ 982us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 03:00:14.323085 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:00:14.323130 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 03:00:14.323221 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:00:14.323753 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:00:14.323761 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:00:14.323768 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 03:00:14.323920 (+ 152us) client_negotiation.cc:240] Negotiation successful
0806 03:00:14.324281 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":281,"threads_started":1}
I20260806 03:00:14.327140 9482 heartbeater.cc:347] Connected to a master server at 127.31.74.254:37151
I20260806 03:00:14.327469 9482 heartbeater.cc:464] Registering TS with master...
I20260806 03:00:14.328294 9482 heartbeater.cc:511] Master 127.31.74.254:37151 requested a full tablet report, sending...
I20260806 03:00:14.329896 9160 ts_manager.cc:194] Registered new tserver with Master: 6dbad88aa22c4681940263ccbcbb0edf (127.31.74.194:40925)
I20260806 03:00:14.330888 9160 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.74.194:50115
W20260806 03:00:14.750360 9486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:00:14.750743 9486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:00:14.750905 9486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:00:14.761354 9486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:00:14.761554 9486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:00:14.761731 9486 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.195
I20260806 03:00:14.772610 9486 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:43383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.74.195
--webserver_port=0
--tserver_master_addrs=127.31.74.254:37151
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.195
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:00:14.774395 9486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:00:14.776731 9486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 03:00:14.780234 9349 heartbeater.cc:503] Master 127.31.74.254:37151 was elected leader, sending a full tablet report...
W20260806 03:00:14.792313 9491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:00:14.792510 9494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:00:14.792606 9492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:00:14.794425 9486 server_base.cc:1034] running on GCE node
I20260806 03:00:14.795152 9486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:00:14.796185 9486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:00:14.797438 9486 hybrid_clock.cc:648] HybridClock initialized: now 1785985214797364 us; error 54 us; skew 500 ppm
I20260806 03:00:14.797966 9486 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 03:00:14.801813 9486 webserver.cc:460] Webserver started at http://127.31.74.195:46171/ using document root <none> and password file <none>
I20260806 03:00:14.803277 9486 fs_manager.cc:359] Metadata directory not provided
I20260806 03:00:14.803520 9486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:00:14.804868 9486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:00:14.808981 9486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-2/data/instance:
uuid: "d158a0f3007d454f9f5a6de6a71940e3"
format_stamp: "Formatted at 2026-08-06 03:00:14 on dist-test-slave-dhph"
server_key: "3fe7032a30e180af6cda552d66699551"
server_key_iv: "035f9804a49c23c84feb859ccec7067a"
server_key_version: "encryptionkey@0"
I20260806 03:00:14.810006 9486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-2/wal/instance:
uuid: "d158a0f3007d454f9f5a6de6a71940e3"
format_stamp: "Formatted at 2026-08-06 03:00:14 on dist-test-slave-dhph"
server_key: "3fe7032a30e180af6cda552d66699551"
server_key_iv: "035f9804a49c23c84feb859ccec7067a"
server_key_version: "encryptionkey@0"
I20260806 03:00:14.817183 9486 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260806 03:00:14.823148 9500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:00:14.825273 9486 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260806 03:00:14.825542 9486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-2/wal
uuid: "d158a0f3007d454f9f5a6de6a71940e3"
format_stamp: "Formatted at 2026-08-06 03:00:14 on dist-test-slave-dhph"
server_key: "3fe7032a30e180af6cda552d66699551"
server_key_iv: "035f9804a49c23c84feb859ccec7067a"
server_key_version: "encryptionkey@0"
I20260806 03:00:14.825832 9486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:00:14.855566 9486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:00:14.856621 9486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:00:14.857206 9486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:00:14.858510 9486 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:00:14.860420 9486 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:00:14.860527 9486 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:00:14.860663 9486 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:00:14.860730 9486 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:00:14.907781 9486 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.195:33207
I20260806 03:00:14.907846 9616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.195:33207 every 8 connection(s)
I20260806 03:00:14.910001 9486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
I20260806 03:00:14.919874 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 9486
I20260806 03:00:14.920085 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-2/wal/instance
I20260806 03:00:14.920789 32043 external_mini_cluster.cc:1428] Setting key 15cd29001acbaa8546f07f074c43bf7b
I20260806 03:00:14.928879 9619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:00:14.913353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.195:51451 (local address 127.31.74.254:37151)
0806 03:00:14.913973 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0806 03:00:14.913980 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 03:00:14.915462 (+ 1482us) server_negotiation.cc:407] Connection header received
0806 03:00:14.917333 (+ 1871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:00:14.917338 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:00:14.917417 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:00:14.917533 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:00:14.919021 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:14.920186 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:14.923188 (+ 3002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:14.924703 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:14.925256 (+ 553us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:14.927393 (+ 2137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:00:14.927399 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:00:14.927402 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:00:14.927425 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:00:14.927484 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:00:14.927498 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:00:14.927576 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 03:00:14.928456 (+ 880us) server_negotiation.cc:299] Negotiation successful
0806 03:00:14.928564 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":233,"threads_started":1}
I20260806 03:00:14.930917 9620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:00:14.914166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:37151 (local address 127.31.74.195:51451)
0806 03:00:14.915257 (+ 1091us) negotiation.cc:107] Waiting for socket to connect
0806 03:00:14.915308 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 03:00:14.917031 (+ 1723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:00:14.917841 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:00:14.917853 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:00:14.917961 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:00:14.918795 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:14.918820 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:14.920453 (+ 1633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:14.920459 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:14.922863 (+ 2404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:14.922884 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:14.925353 (+ 2469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:14.925381 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:14.925643 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:14.926815 (+ 1172us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 03:00:14.926819 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:00:14.926890 (+ 71us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 03:00:14.927059 (+ 169us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:00:14.927795 (+ 736us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:00:14.927815 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:00:14.927827 (+ 12us) client_negotiation.cc:769] Sending connection context
0806 03:00:14.928356 (+ 529us) client_negotiation.cc:240] Negotiation successful
0806 03:00:14.929072 (+ 716us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":358,"threads_started":1}
I20260806 03:00:14.933665 9617 heartbeater.cc:347] Connected to a master server at 127.31.74.254:37151
I20260806 03:00:14.934094 9617 heartbeater.cc:464] Registering TS with master...
I20260806 03:00:14.935163 9617 heartbeater.cc:511] Master 127.31.74.254:37151 requested a full tablet report, sending...
I20260806 03:00:14.937655 9160 ts_manager.cc:194] Registered new tserver with Master: d158a0f3007d454f9f5a6de6a71940e3 (127.31.74.195:33207)
I20260806 03:00:14.940176 9160 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.74.195:51451
I20260806 03:00:14.942214 32043 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260806 03:00:14.951547 9619 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:00:14.949542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59914 (local address 127.31.74.254:37151)
0806 03:00:14.949977 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0806 03:00:14.949986 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 03:00:14.950324 (+ 338us) server_negotiation.cc:407] Connection header received
0806 03:00:14.950570 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:00:14.950617 (+ 47us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:00:14.950703 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:00:14.950829 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:00:14.951343 (+ 514us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59914: BlockingRecv error: recv got EOF from 127.0.0.1:59914 (error 108)
Metrics: {"server-negotiator.queue_time_us":153}
I20260806 03:00:14.953367 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 9220
I20260806 03:00:14.965948 9627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:00:14.955250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47890 (local address 127.31.74.193:45433)
0806 03:00:14.955985 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0806 03:00:14.955993 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 03:00:14.956025 (+ 32us) server_negotiation.cc:407] Connection header received
0806 03:00:14.956088 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:00:14.956094 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:00:14.956330 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:00:14.956627 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:00:14.957095 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:14.958440 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:14.960657 (+ 2217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:14.961426 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:14.961595 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:14.963609 (+ 2014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:00:14.963620 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:00:14.963636 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:00:14.964482 (+ 846us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:00:14.964548 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:00:14.964575 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:00:14.964715 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0806 03:00:14.965404 (+ 689us) server_negotiation.cc:299] Negotiation successful
0806 03:00:14.965547 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":396,"threads_started":1}
I20260806 03:00:14.968696 9344 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 03:00:15.068112 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 9220
I20260806 03:00:15.090226 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 9353
I20260806 03:00:15.100457 9629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:00:15.091588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46490 (local address 127.31.74.194:40925)
0806 03:00:15.092216 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0806 03:00:15.092223 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 03:00:15.092255 (+ 32us) server_negotiation.cc:407] Connection header received
0806 03:00:15.092319 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:00:15.092326 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:00:15.092590 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:00:15.092731 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:00:15.093222 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:15.094408 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:15.096303 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:15.096970 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:15.097102 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:15.098423 (+ 1321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:00:15.098435 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:00:15.098564 (+ 129us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:00:15.099369 (+ 805us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:00:15.099424 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:00:15.099451 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:00:15.099558 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 03:00:15.099863 (+ 305us) server_negotiation.cc:299] Negotiation successful
0806 03:00:15.100069 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":298,"threads_started":1}
I20260806 03:00:15.101986 9477 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 03:00:15.193748 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 9353
I20260806 03:00:15.213829 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 9486
I20260806 03:00:15.223270 9631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:00:15.215001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32786 (local address 127.31.74.195:33207)
0806 03:00:15.215707 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0806 03:00:15.215714 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 03:00:15.215745 (+ 31us) server_negotiation.cc:407] Connection header received
0806 03:00:15.215836 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:00:15.215843 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:00:15.216068 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:00:15.216198 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:00:15.216642 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:15.217797 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:15.219361 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:15.220022 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:15.220139 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:15.221705 (+ 1566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:00:15.221714 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:00:15.221727 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:00:15.222354 (+ 627us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:00:15.222403 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:00:15.222423 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:00:15.222517 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0806 03:00:15.222802 (+ 285us) server_negotiation.cc:299] Negotiation successful
0806 03:00:15.222924 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":392,"threads_started":1}
I20260806 03:00:15.224908 9612 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 03:00:15.314950 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 9486
I20260806 03:00:15.335960 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 9128
I20260806 03:00:15.337312 9190 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 03:00:15.445778 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 9128
2026-08-06T03:00:15Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (3039 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-06T03:00:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:00:15Z Disabled control of system clock
I20260806 03:00:15.556512 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785985105255475-32043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785985105255475-32043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785985105255475-32043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:43077
--webserver_interface=127.31.74.254
--webserver_port=0
--builtin_ntp_servers=127.31.74.212:37989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.74.254:43077
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260806 03:00:15.995363 9637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:00:15.995723 9637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:00:15.995978 9637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:00:16.004758 9637 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:00:16.004928 9637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:00:16.004976 9637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:00:16.005062 9637 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:00:16.005156 9637 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:00:16.019024 9637 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:37989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785985105255475-32043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785985105255475-32043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.74.254:43077
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:43077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.74.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785985105255475-32043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:00:16.021265 9637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:00:16.023389 9637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:00:16.049722 9643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:00:16.050480 9642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:00:16.050638 9645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:00:16.054543 9637 server_base.cc:1034] running on GCE node
I20260806 03:00:16.056124 9637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:00:16.059497 9637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:00:16.061056 9637 hybrid_clock.cc:648] HybridClock initialized: now 1785985216060980 us; error 68 us; skew 500 ppm
I20260806 03:00:16.061931 9637 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 03:00:16.072171 9637 webserver.cc:460] Webserver started at http://127.31.74.254:39073/ using document root <none> and password file <none>
I20260806 03:00:16.073860 9637 fs_manager.cc:359] Metadata directory not provided
I20260806 03:00:16.074083 9637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:00:16.075026 9637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:00:16.081576 9637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785985105255475-32043-0/minicluster-data/master-0/data/instance:
uuid: "259256c7e5a2485abf4894bd5db0ea21"
format_stamp: "Formatted at 2026-08-06 03:00:16 on dist-test-slave-dhph"
server_key: "c624326171d9870e3b38faff82fe7b94"
server_key_iv: "64fd323bceddaa16bfd9bcdb80513a83"
server_key_version: "encryptionkey@0"
I20260806 03:00:16.083858 9637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785985105255475-32043-0/minicluster-data/master-0/wal/instance:
uuid: "259256c7e5a2485abf4894bd5db0ea21"
format_stamp: "Formatted at 2026-08-06 03:00:16 on dist-test-slave-dhph"
server_key: "c624326171d9870e3b38faff82fe7b94"
server_key_iv: "64fd323bceddaa16bfd9bcdb80513a83"
server_key_version: "encryptionkey@0"
I20260806 03:00:16.096141 9637 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.002s sys 0.009s
I20260806 03:00:16.101879 9651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:00:16.104888 9637 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.001s
I20260806 03:00:16.105297 9637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785985105255475-32043-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785985105255475-32043-0/minicluster-data/master-0/wal
uuid: "259256c7e5a2485abf4894bd5db0ea21"
format_stamp: "Formatted at 2026-08-06 03:00:16 on dist-test-slave-dhph"
server_key: "c624326171d9870e3b38faff82fe7b94"
server_key_iv: "64fd323bceddaa16bfd9bcdb80513a83"
server_key_version: "encryptionkey@0"
I20260806 03:00:16.106125 9637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785985105255475-32043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785985105255475-32043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785985105255475-32043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:00:16.118553 9637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:00:16.120024 9637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:00:16.120674 9637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:00:16.161360 9637 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.254:43077
I20260806 03:00:16.161394 9706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.254:43077 every 8 connection(s)
I20260806 03:00:16.164557 9637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
I20260806 03:00:16.167387 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 9637
I20260806 03:00:16.167843 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785985105255475-32043-0/minicluster-data/master-0/wal/instance
I20260806 03:00:16.169456 32043 external_mini_cluster.cc:1428] Setting key ec0e184b5bf3ad241112d0d5a8d451be
I20260806 03:00:16.173528 9707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:00:16.186309 9711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:00:16.174235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56258 (local address 127.31.74.254:43077)
0806 03:00:16.176172 (+ 1937us) server_negotiation.cc:206] Beginning negotiation
0806 03:00:16.176184 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 03:00:16.176233 (+ 49us) server_negotiation.cc:407] Connection header received
0806 03:00:16.178097 (+ 1864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:00:16.178122 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:00:16.178761 (+ 639us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:00:16.179426 (+ 665us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:00:16.181868 (+ 2442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:00:16.181896 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:00:16.181939 (+ 43us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:00:16.181998 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:00:16.182208 (+ 210us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:00:16.182244 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:00:16.183060 (+ 816us) server_negotiation.cc:1035] Waiting for connection context
0806 03:00:16.183335 (+ 275us) server_negotiation.cc:299] Negotiation successful
0806 03:00:16.184010 (+ 675us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1146,"thread_start_us":828,"threads_started":1}
I20260806 03:00:16.196015 9707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 259256c7e5a2485abf4894bd5db0ea21: Bootstrap starting.
I20260806 03:00:16.202863 9707 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 259256c7e5a2485abf4894bd5db0ea21: Neither blocks nor log segments found. Creating new log.
I20260806 03:00:16.205536 9707 log.cc:824] T 00000000000000000000000000000000 P 259256c7e5a2485abf4894bd5db0ea21: Log is configured to *not* fsync() on all Append() calls
I20260806 03:00:16.211576 9707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 259256c7e5a2485abf4894bd5db0ea21: No bootstrap required, opened a new log
I20260806 03:00:16.227267 9707 raft_consensus.cc:348] T 00000000000000000000000000000000 P 259256c7e5a2485abf4894bd5db0ea21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "259256c7e5a2485abf4894bd5db0ea21" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 43077 } }
I20260806 03:00:16.228058 9707 raft_consensus.cc:374] T 00000000000000000000000000000000 P 259256c7e5a2485abf4894bd5db0ea21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:00:16.228220 9707 raft_consensus.cc:729] T 00000000000000000000000000000000 P 259256c7e5a2485abf4894bd5db0ea21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 259256c7e5a2485abf4894bd5db0ea21, State: Initialized, Role: FOLLOWER
I20260806 03:00:16.229357 9707 consensus_queue.cc:260] T 00000000000000000000000000000000 P 259256c7e5a2485abf4894bd5db0ea21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "259256c7e5a2485abf4894bd5db0ea21" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 43077 } }
I20260806 03:00:16.229837 9707 raft_consensus.cc:388] T 00000000000000000000000000000000 P 259256c7e5a2485abf4894bd5db0ea21 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:00:16.230350 9707 raft_consensus.cc:482] T 00000000000000000000000000000000 P 259256c7e5a2485abf4894bd5db0ea21 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:00:16.231220 9707 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 259256c7e5a2485abf4894bd5db0ea21 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:00:16.236645 9707 raft_consensus.cc:504] T 00000000000000000000000000000000 P 259256c7e5a2485abf4894bd5db0ea21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "259256c7e5a2485abf4894bd5db0ea21" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 43077 } }
I20260806 03:00:16.238175 9707 leader_election.cc:302] T 00000000000000000000000000000000 P 259256c7e5a2485abf4894bd5db0ea21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 259256c7e5a2485abf4894bd5db0ea21; no voters:
I20260806 03:00:16.239485 9707 leader_election.cc:288] T 00000000000000000000000000000000 P 259256c7e5a2485abf4894bd5db0ea21 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:00:16.239665 9713 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 259256c7e5a2485abf4894bd5db0ea21 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:00:16.241951 9713 raft_consensus.cc:686] T 00000000000000000000000000000000 P 259256c7e5a2485abf4894bd5db0ea21 [term 1 LEADER]: Becoming Leader. State: Replica: 259256c7e5a2485abf4894bd5db0ea21, State: Running, Role: LEADER
I20260806 03:00:16.243363 9713 consensus_queue.cc:237] T 00000000000000000000000000000000 P 259256c7e5a2485abf4894bd5db0ea21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "259256c7e5a2485abf4894bd5db0ea21" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 43077 } }
I20260806 03:00:16.244122 9707 sys_catalog.cc:564] T 00000000000000000000000000000000 P 259256c7e5a2485abf4894bd5db0ea21 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:00:16.253314 9714 sys_catalog.cc:455] T 00000000000000000000000000000000 P 259256c7e5a2485abf4894bd5db0ea21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "259256c7e5a2485abf4894bd5db0ea21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "259256c7e5a2485abf4894bd5db0ea21" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 43077 } } }
I20260806 03:00:16.255003 9715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 259256c7e5a2485abf4894bd5db0ea21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 259256c7e5a2485abf4894bd5db0ea21. Latest consensus state: current_term: 1 leader_uuid: "259256c7e5a2485abf4894bd5db0ea21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "259256c7e5a2485abf4894bd5db0ea21" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 43077 } } }
I20260806 03:00:16.256775 9715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 259256c7e5a2485abf4894bd5db0ea21 [sys.catalog]: This master's current role is: LEADER
I20260806 03:00:16.259166 9714 sys_catalog.cc:458] T 00000000000000000000000000000000 P 259256c7e5a2485abf4894bd5db0ea21 [sys.catalog]: This master's current role is: LEADER
I20260806 03:00:16.259737 9722 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:00:16.289127 9722 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:00:16.311923 9722 catalog_manager.cc:1347] Generated new cluster ID: bbe6fd57a81b4cf49a245b9b446415ad
I20260806 03:00:16.312071 9722 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:00:16.324095 9722 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:00:16.324674 9722 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:00:16.352859 9722 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 259256c7e5a2485abf4894bd5db0ea21: Generated new TSK 0
I20260806 03:00:16.354373 9722 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 03:00:16.371937 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.193:0
--local_ip_for_outbound_sockets=127.31.74.193
--webserver_interface=127.31.74.193
--webserver_port=0
--tserver_master_addrs=127.31.74.254:43077
--builtin_ntp_servers=127.31.74.212:37989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260806 03:00:16.809677 9732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:00:16.810446 9732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:00:16.810911 9732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:00:16.821947 9732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:00:16.822129 9732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:00:16.822361 9732 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.193
I20260806 03:00:16.835183 9732 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:37989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.74.193
--webserver_port=0
--tserver_master_addrs=127.31.74.254:43077
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.193
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:00:16.836692 9732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:00:16.839239 9732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:00:16.856003 9737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:00:16.856303 9738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:00:16.856431 9740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:00:16.858031 9732 server_base.cc:1034] running on GCE node
I20260806 03:00:16.859750 9732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:00:16.861547 9732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:00:16.863061 9732 hybrid_clock.cc:648] HybridClock initialized: now 1785985216862973 us; error 74 us; skew 500 ppm
I20260806 03:00:16.863677 9732 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 03:00:16.867333 9732 webserver.cc:460] Webserver started at http://127.31.74.193:45435/ using document root <none> and password file <none>
I20260806 03:00:16.868209 9732 fs_manager.cc:359] Metadata directory not provided
I20260806 03:00:16.868363 9732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:00:16.868746 9732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:00:16.872437 9732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-0/data/instance:
uuid: "2a33577773f644d185cb988f846844d1"
format_stamp: "Formatted at 2026-08-06 03:00:16 on dist-test-slave-dhph"
server_key: "c4e3df074ab3113cd093ab47ab7908f5"
server_key_iv: "f21af5c73de6198ac7509bef68dd1f17"
server_key_version: "encryptionkey@0"
I20260806 03:00:16.873625 9732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-0/wal/instance:
uuid: "2a33577773f644d185cb988f846844d1"
format_stamp: "Formatted at 2026-08-06 03:00:16 on dist-test-slave-dhph"
server_key: "c4e3df074ab3113cd093ab47ab7908f5"
server_key_iv: "f21af5c73de6198ac7509bef68dd1f17"
server_key_version: "encryptionkey@0"
I20260806 03:00:16.881213 9732 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260806 03:00:16.886189 9746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:00:16.888475 9732 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 03:00:16.888971 9732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-0/wal
uuid: "2a33577773f644d185cb988f846844d1"
format_stamp: "Formatted at 2026-08-06 03:00:16 on dist-test-slave-dhph"
server_key: "c4e3df074ab3113cd093ab47ab7908f5"
server_key_iv: "f21af5c73de6198ac7509bef68dd1f17"
server_key_version: "encryptionkey@0"
I20260806 03:00:16.889586 9732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:00:16.897095 9732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:00:16.898540 9732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:00:16.899049 9732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:00:16.900600 9732 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:00:16.902880 9732 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:00:16.903057 9732 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:00:16.903223 9732 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:00:16.903324 9732 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:00:16.953614 9732 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.193:46147
I20260806 03:00:16.953799 9861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.193:46147 every 8 connection(s)
I20260806 03:00:16.955128 9732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
I20260806 03:00:16.963236 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 9732
I20260806 03:00:16.963552 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-0/wal/instance
I20260806 03:00:16.964434 32043 external_mini_cluster.cc:1428] Setting key eec9f52d60993b16fab9816d815322df
I20260806 03:00:16.965796 9864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:00:16.958060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.193:42077 (local address 127.31.74.254:43077)
0806 03:00:16.958789 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0806 03:00:16.958795 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 03:00:16.959971 (+ 1176us) server_negotiation.cc:407] Connection header received
0806 03:00:16.962065 (+ 2094us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:00:16.962070 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:00:16.962239 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:00:16.962510 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:00:16.964441 (+ 1931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:00:16.964447 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:00:16.964452 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:00:16.964480 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:00:16.964529 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:00:16.964542 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:00:16.964702 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0806 03:00:16.965222 (+ 520us) server_negotiation.cc:299] Negotiation successful
0806 03:00:16.965302 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":346,"threads_started":1}
I20260806 03:00:16.967612 9865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:00:16.958514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:43077 (local address 127.31.74.193:42077)
0806 03:00:16.959662 (+ 1148us) negotiation.cc:107] Waiting for socket to connect
0806 03:00:16.959712 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 03:00:16.961708 (+ 1996us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:00:16.962900 (+ 1192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:00:16.962911 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:00:16.963018 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:00:16.964050 (+ 1032us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 03:00:16.964052 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:00:16.964098 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 03:00:16.964223 (+ 125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:00:16.964753 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:00:16.964778 (+ 25us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:00:16.964793 (+ 15us) client_negotiation.cc:769] Sending connection context
0806 03:00:16.965299 (+ 506us) client_negotiation.cc:240] Negotiation successful
0806 03:00:16.965710 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":346,"threads_started":1}
I20260806 03:00:16.968717 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.194:0
--local_ip_for_outbound_sockets=127.31.74.194
--webserver_interface=127.31.74.194
--webserver_port=0
--tserver_master_addrs=127.31.74.254:43077
--builtin_ntp_servers=127.31.74.212:37989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260806 03:00:16.970136 9862 heartbeater.cc:347] Connected to a master server at 127.31.74.254:43077
I20260806 03:00:16.970536 9862 heartbeater.cc:464] Registering TS with master...
I20260806 03:00:16.971465 9862 heartbeater.cc:511] Master 127.31.74.254:43077 requested a full tablet report, sending...
I20260806 03:00:16.974118 9668 ts_manager.cc:194] Registered new tserver with Master: 2a33577773f644d185cb988f846844d1 (127.31.74.193:46147)
W20260806 03:00:17.380085 9866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:00:17.380482 9866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:00:17.380661 9866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:00:17.388871 9866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:00:17.389086 9866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:00:17.389279 9866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.194
I20260806 03:00:17.399924 9866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:37989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.74.194
--webserver_port=0
--tserver_master_addrs=127.31.74.254:43077
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.194
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:00:17.401405 9866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:00:17.403263 9866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:00:17.418998 9871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:00:17.418970 9872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:00:17.418993 9874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:00:17.420435 9866 server_base.cc:1034] running on GCE node
I20260806 03:00:17.421387 9866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:00:17.422472 9866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:00:17.423774 9866 hybrid_clock.cc:648] HybridClock initialized: now 1785985217423697 us; error 55 us; skew 500 ppm
I20260806 03:00:17.424171 9866 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 03:00:17.427048 9866 webserver.cc:460] Webserver started at http://127.31.74.194:34357/ using document root <none> and password file <none>
I20260806 03:00:17.427862 9866 fs_manager.cc:359] Metadata directory not provided
I20260806 03:00:17.428013 9866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:00:17.428385 9866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:00:17.431679 9866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-1/data/instance:
uuid: "5cd5cd233eff4e76b981f8b8e981dc8d"
format_stamp: "Formatted at 2026-08-06 03:00:17 on dist-test-slave-dhph"
server_key: "0493ebd0a0a1c48d6d4e0a047cfbafd2"
server_key_iv: "ecbc95604c49e175b8c2bfda29127ba4"
server_key_version: "encryptionkey@0"
I20260806 03:00:17.432627 9866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-1/wal/instance:
uuid: "5cd5cd233eff4e76b981f8b8e981dc8d"
format_stamp: "Formatted at 2026-08-06 03:00:17 on dist-test-slave-dhph"
server_key: "0493ebd0a0a1c48d6d4e0a047cfbafd2"
server_key_iv: "ecbc95604c49e175b8c2bfda29127ba4"
server_key_version: "encryptionkey@0"
I20260806 03:00:17.439298 9866 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260806 03:00:17.444463 9880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:00:17.446497 9866 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260806 03:00:17.446792 9866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-1/wal
uuid: "5cd5cd233eff4e76b981f8b8e981dc8d"
format_stamp: "Formatted at 2026-08-06 03:00:17 on dist-test-slave-dhph"
server_key: "0493ebd0a0a1c48d6d4e0a047cfbafd2"
server_key_iv: "ecbc95604c49e175b8c2bfda29127ba4"
server_key_version: "encryptionkey@0"
I20260806 03:00:17.447072 9866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:00:17.454876 9866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:00:17.455912 9866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:00:17.456254 9866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:00:17.457617 9866 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:00:17.459466 9866 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:00:17.459609 9866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:00:17.459750 9866 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:00:17.459857 9866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:00:17.503888 9866 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.194:46365
I20260806 03:00:17.503994 9996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.194:46365 every 8 connection(s)
I20260806 03:00:17.505417 9866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
I20260806 03:00:17.507077 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 9866
I20260806 03:00:17.507385 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-1/wal/instance
I20260806 03:00:17.508144 32043 external_mini_cluster.cc:1428] Setting key 2eb9c1fa8a8beea74764202e56d185f8
I20260806 03:00:17.512012 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.195:0
--local_ip_for_outbound_sockets=127.31.74.195
--webserver_interface=127.31.74.195
--webserver_port=0
--tserver_master_addrs=127.31.74.254:43077
--builtin_ntp_servers=127.31.74.212:37989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260806 03:00:17.516330 9999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:00:17.509528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.194:43849 (local address 127.31.74.254:43077)
0806 03:00:17.510032 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0806 03:00:17.510038 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 03:00:17.511894 (+ 1856us) server_negotiation.cc:407] Connection header received
0806 03:00:17.513723 (+ 1829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:00:17.513729 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:00:17.513802 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:00:17.513934 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:00:17.515434 (+ 1500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:00:17.515441 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:00:17.515446 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:00:17.515467 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:00:17.515555 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:00:17.515574 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:00:17.515740 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0806 03:00:17.515933 (+ 193us) server_negotiation.cc:299] Negotiation successful
0806 03:00:17.516006 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":271,"threads_started":1}
I20260806 03:00:17.517309 10000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:00:17.510482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:43077 (local address 127.31.74.194:43849)
0806 03:00:17.511673 (+ 1191us) negotiation.cc:107] Waiting for socket to connect
0806 03:00:17.511751 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0806 03:00:17.513430 (+ 1679us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:00:17.514188 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:00:17.514198 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:00:17.514299 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:00:17.515112 (+ 813us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 03:00:17.515115 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:00:17.515162 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 03:00:17.515255 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:00:17.515702 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:00:17.515713 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:00:17.515721 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 03:00:17.516023 (+ 302us) client_negotiation.cc:240] Negotiation successful
0806 03:00:17.516299 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"thread_start_us":396,"threads_started":1}
I20260806 03:00:17.519049 9997 heartbeater.cc:347] Connected to a master server at 127.31.74.254:43077
I20260806 03:00:17.519418 9997 heartbeater.cc:464] Registering TS with master...
I20260806 03:00:17.520193 9997 heartbeater.cc:511] Master 127.31.74.254:43077 requested a full tablet report, sending...
I20260806 03:00:17.522188 9668 ts_manager.cc:194] Registered new tserver with Master: 5cd5cd233eff4e76b981f8b8e981dc8d (127.31.74.194:46365)
W20260806 03:00:17.909799 10001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:00:17.910159 10001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:00:17.910267 10001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:00:17.919641 10001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:00:17.919778 10001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:00:17.919896 10001 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.195
I20260806 03:00:17.929576 10001 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:37989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.74.195
--webserver_port=0
--tserver_master_addrs=127.31.74.254:43077
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.195
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:00:17.930966 10001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:00:17.932729 10001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:00:17.946768 10006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:00:17.946676 10010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:00:17.946722 10007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:00:17.947989 10001 server_base.cc:1034] running on GCE node
I20260806 03:00:17.948625 10001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:00:17.949661 10001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:00:17.950934 10001 hybrid_clock.cc:648] HybridClock initialized: now 1785985217950880 us; error 46 us; skew 500 ppm
I20260806 03:00:17.951399 10001 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 03:00:17.954962 10001 webserver.cc:460] Webserver started at http://127.31.74.195:46649/ using document root <none> and password file <none>
I20260806 03:00:17.955761 10001 fs_manager.cc:359] Metadata directory not provided
I20260806 03:00:17.955914 10001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:00:17.956333 10001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:00:17.959630 10001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-2/data/instance:
uuid: "3f5a90e11cf14c34a73d4f25a3b095ce"
format_stamp: "Formatted at 2026-08-06 03:00:17 on dist-test-slave-dhph"
server_key: "d97a18bb1749f117faaa8fbc6deb55c2"
server_key_iv: "8de80c6d215478d72e2e76f12bacc192"
server_key_version: "encryptionkey@0"
I20260806 03:00:17.960487 10001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-2/wal/instance:
uuid: "3f5a90e11cf14c34a73d4f25a3b095ce"
format_stamp: "Formatted at 2026-08-06 03:00:17 on dist-test-slave-dhph"
server_key: "d97a18bb1749f117faaa8fbc6deb55c2"
server_key_iv: "8de80c6d215478d72e2e76f12bacc192"
server_key_version: "encryptionkey@0"
I20260806 03:00:17.967089 10001 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260806 03:00:17.971719 10016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:00:17.973407 10001 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260806 03:00:17.973719 10001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-2/wal
uuid: "3f5a90e11cf14c34a73d4f25a3b095ce"
format_stamp: "Formatted at 2026-08-06 03:00:17 on dist-test-slave-dhph"
server_key: "d97a18bb1749f117faaa8fbc6deb55c2"
server_key_iv: "8de80c6d215478d72e2e76f12bacc192"
server_key_version: "encryptionkey@0"
I20260806 03:00:17.973982 10001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:00:17.977859 9862 heartbeater.cc:503] Master 127.31.74.254:43077 was elected leader, sending a full tablet report...
I20260806 03:00:17.981449 10001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:00:17.982368 10001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:00:17.982753 10001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:00:17.983749 10001 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:00:17.985332 10001 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:00:17.985430 10001 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:00:17.985517 10001 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:00:17.985581 10001 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:00:18.029623 10001 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.195:43303
I20260806 03:00:18.029860 10134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.195:43303 every 8 connection(s)
I20260806 03:00:18.031251 10001 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
I20260806 03:00:18.039657 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 10001
I20260806 03:00:18.039898 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785985105255475-32043-0/minicluster-data/ts-2/wal/instance
I20260806 03:00:18.040488 32043 external_mini_cluster.cc:1428] Setting key f35032913d63db3dd080a59647c17fe8
I20260806 03:00:18.040599 10137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:00:18.034217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.195:39053 (local address 127.31.74.254:43077)
0806 03:00:18.034796 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0806 03:00:18.034804 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 03:00:18.036318 (+ 1514us) server_negotiation.cc:407] Connection header received
0806 03:00:18.038034 (+ 1716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:00:18.038039 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:00:18.038096 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:00:18.038227 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:00:18.039695 (+ 1468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:00:18.039703 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:00:18.039709 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:00:18.039732 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:00:18.039793 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:00:18.039812 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:00:18.039968 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0806 03:00:18.040168 (+ 200us) server_negotiation.cc:299] Negotiation successful
0806 03:00:18.040246 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":249,"threads_started":1}
I20260806 03:00:18.042045 10138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:00:18.034878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:43077 (local address 127.31.74.195:39053)
0806 03:00:18.036129 (+ 1251us) negotiation.cc:107] Waiting for socket to connect
0806 03:00:18.036174 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 03:00:18.037753 (+ 1579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:00:18.038466 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:00:18.038475 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:00:18.038575 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:00:18.039368 (+ 793us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 03:00:18.039371 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:00:18.039409 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 03:00:18.039495 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:00:18.039924 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:00:18.039932 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:00:18.039938 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 03:00:18.040221 (+ 283us) client_negotiation.cc:240] Negotiation successful
0806 03:00:18.040649 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":318,"threads_started":1}
I20260806 03:00:18.043671 10135 heartbeater.cc:347] Connected to a master server at 127.31.74.254:43077
I20260806 03:00:18.044045 10135 heartbeater.cc:464] Registering TS with master...
I20260806 03:00:18.044746 10135 heartbeater.cc:511] Master 127.31.74.254:43077 requested a full tablet report, sending...
I20260806 03:00:18.046840 9668 ts_manager.cc:194] Registered new tserver with Master: 3f5a90e11cf14c34a73d4f25a3b095ce (127.31.74.195:43303)
I20260806 03:00:18.057415 32043 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260806 03:00:18.063975 10137 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:00:18.062422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50286 (local address 127.31.74.254:43077)
0806 03:00:18.062738 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0806 03:00:18.062744 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 03:00:18.063130 (+ 386us) server_negotiation.cc:407] Connection header received
0806 03:00:18.063269 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:00:18.063274 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:00:18.063329 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:00:18.063425 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:00:18.063763 (+ 338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50286: BlockingRecv error: recv got EOF from 127.0.0.1:50286 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260806 03:00:18.064811 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 9732
I20260806 03:00:18.070194 10145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:00:18.067182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55076 (local address 127.31.74.193:46147)
0806 03:00:18.067810 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0806 03:00:18.067818 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 03:00:18.067843 (+ 25us) server_negotiation.cc:407] Connection header received
0806 03:00:18.067912 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:00:18.067919 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:00:18.068130 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:00:18.068314 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:00:18.069442 (+ 1128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:00:18.069448 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:00:18.069459 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:00:18.069471 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:00:18.069508 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:00:18.069522 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:00:18.069723 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0806 03:00:18.069791 (+ 68us) server_negotiation.cc:299] Negotiation successful
0806 03:00:18.069896 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":304,"threads_started":1}
I20260806 03:00:18.071663 9857 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 03:00:18.165000 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 9732
I20260806 03:00:18.191674 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 9866
I20260806 03:00:18.195856 10147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:00:18.192698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51164 (local address 127.31.74.194:46365)
0806 03:00:18.193288 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0806 03:00:18.193297 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 03:00:18.193324 (+ 27us) server_negotiation.cc:407] Connection header received
0806 03:00:18.193394 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:00:18.193400 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:00:18.193644 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:00:18.193890 (+ 246us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:00:18.195040 (+ 1150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:00:18.195046 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:00:18.195058 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:00:18.195078 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:00:18.195120 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:00:18.195135 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:00:18.195298 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0806 03:00:18.195435 (+ 137us) server_negotiation.cc:299] Negotiation successful
0806 03:00:18.195557 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":280,"threads_started":1}
I20260806 03:00:18.197230 9992 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 03:00:18.288481 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 9866
I20260806 03:00:18.309880 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 10001
I20260806 03:00:18.314664 10149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:00:18.311104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46258 (local address 127.31.74.195:43303)
0806 03:00:18.311828 (+ 724us) server_negotiation.cc:206] Beginning negotiation
0806 03:00:18.311835 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 03:00:18.311867 (+ 32us) server_negotiation.cc:407] Connection header received
0806 03:00:18.311932 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:00:18.311939 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:00:18.312167 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:00:18.312369 (+ 202us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:00:18.313563 (+ 1194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:00:18.313570 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:00:18.313580 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:00:18.313597 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:00:18.313646 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:00:18.313660 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:00:18.314004 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0806 03:00:18.314110 (+ 106us) server_negotiation.cc:299] Negotiation successful
0806 03:00:18.314229 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":376,"threads_started":1}
I20260806 03:00:18.316174 10130 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 03:00:18.405483 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 10001
I20260806 03:00:18.426946 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 9637
I20260806 03:00:18.428215 9701 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 03:00:18.538887 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 9637
2026-08-06T03:00:18Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (3089 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 03:00:18 dist-test-slave-dhph krb5kdc[10155](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 03:00:18 dist-test-slave-dhph krb5kdc[10155](info): set up 2 sockets
Aug 06 03:00:18 dist-test-slave-dhph krb5kdc[10155](info): commencing operation
krb5kdc: starting...
W20260806 03:00:19.798506 32043 mini_kdc.cc:121] Time spent starting KDC: real 1.212s user 0.000s sys 0.053s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/krb5kdc/test-user.keytab.
Aug 06 03:00:20 dist-test-slave-dhph krb5kdc[10155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T03:00:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:00:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.74.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab.
Entry for principal kudu/127.31.74.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab.
I20260806 03:00:20.160719 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:45429
--webserver_interface=127.31.74.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab
--principal=kudu/127.31.74.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:44985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.74.254:45429
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:00:20.541561 10168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:00:20.541879 10168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:00:20.542030 10168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:00:20.549715 10168 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:00:20.549842 10168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:00:20.549921 10168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:00:20.550009 10168 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:00:20.550050 10168 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:00:20.560742 10168 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:44985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.74.254:45429
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab
--principal=kudu/127.31.74.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:45429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.74.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:00:20.562152 10168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:00:20.563520 10168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:00:20.576510 10173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:00:20.576553 10176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:00:20.576586 10174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:00:20.577461 10168 server_base.cc:1034] running on GCE node
I20260806 03:00:20.578347 10168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:00:20.580026 10168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:00:20.581301 10168 hybrid_clock.cc:648] HybridClock initialized: now 1785985220581236 us; error 53 us; skew 500 ppm
I20260806 03:00:20.581764 10168 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 03:00:20 dist-test-slave-dhph krb5kdc[10155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985220, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:00:20.585780 10168 init.cc:377] Logged in from keytab as kudu/127.31.74.254@KRBTEST.COM (short username kudu)
I20260806 03:00:20.587602 10168 webserver.cc:460] Webserver started at http://127.31.74.254:37607/ using document root <none> and password file <none>
I20260806 03:00:20.588337 10168 fs_manager.cc:359] Metadata directory not provided
I20260806 03:00:20.588451 10168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:00:20.588933 10168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:00:20.592231 10168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/minicluster-data/master-0/data/instance:
uuid: "97ed33cf1323474e9dda4620b71416df"
format_stamp: "Formatted at 2026-08-06 03:00:20 on dist-test-slave-dhph"
server_key: "6ee5a1df8242f1dcb2124ae681047bb9"
server_key_iv: "475a680b65a3709bf128b041fdf82f85"
server_key_version: "encryptionkey@0"
I20260806 03:00:20.593056 10168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/minicluster-data/master-0/wal/instance:
uuid: "97ed33cf1323474e9dda4620b71416df"
format_stamp: "Formatted at 2026-08-06 03:00:20 on dist-test-slave-dhph"
server_key: "6ee5a1df8242f1dcb2124ae681047bb9"
server_key_iv: "475a680b65a3709bf128b041fdf82f85"
server_key_version: "encryptionkey@0"
I20260806 03:00:20.600955 10168 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.003s
I20260806 03:00:20.605397 10183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:00:20.607161 10168 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 03:00:20.607375 10168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/minicluster-data/master-0/wal
uuid: "97ed33cf1323474e9dda4620b71416df"
format_stamp: "Formatted at 2026-08-06 03:00:20 on dist-test-slave-dhph"
server_key: "6ee5a1df8242f1dcb2124ae681047bb9"
server_key_iv: "475a680b65a3709bf128b041fdf82f85"
server_key_version: "encryptionkey@0"
I20260806 03:00:20.607681 10168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:00:20.620100 10168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:00:20.624756 10168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:00:20.625124 10168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:00:20.647543 10168 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.254:45429
I20260806 03:00:20.647665 10234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.254:45429 every 8 connection(s)
I20260806 03:00:20.649050 10168 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
I20260806 03:00:20.653062 10235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:00:20.656250 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 10168
I20260806 03:00:20.656553 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/minicluster-data/master-0/wal/instance
I20260806 03:00:20.657298 32043 external_mini_cluster.cc:1428] Setting key 44cf8bf5a868dbf6983860ccab2e5193
I20260806 03:00:20.668139 10235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97ed33cf1323474e9dda4620b71416df: Bootstrap starting.
I20260806 03:00:20.673173 10235 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 97ed33cf1323474e9dda4620b71416df: Neither blocks nor log segments found. Creating new log.
I20260806 03:00:20.674666 10235 log.cc:824] T 00000000000000000000000000000000 P 97ed33cf1323474e9dda4620b71416df: Log is configured to *not* fsync() on all Append() calls
Aug 06 03:00:20 dist-test-slave-dhph krb5kdc[10155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 03:00:20.678670 10235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97ed33cf1323474e9dda4620b71416df: No bootstrap required, opened a new log
I20260806 03:00:20.684913 10238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:00:20.659698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34348 (local address 127.31.74.254:45429)
0806 03:00:20.660658 (+ 960us) server_negotiation.cc:206] Beginning negotiation
0806 03:00:20.660670 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 03:00:20.660733 (+ 63us) server_negotiation.cc:407] Connection header received
0806 03:00:20.662523 (+ 1790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:00:20.662545 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:00:20.663042 (+ 497us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:00:20.663397 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:00:20.665891 (+ 2494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:20.667813 (+ 1922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:20.670005 (+ 2192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:20.671065 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:20.671237 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:20.676918 (+ 5681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:00:20.676928 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:00:20.676945 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:00:20.676987 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:00:20.680453 (+ 3466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:00:20.681263 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:00:20.681271 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:00:20.681278 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:00:20.681462 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:00:20.681859 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:00:20.681864 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:00:20.681873 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:00:20.682449 (+ 576us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:00:20.682827 (+ 378us) server_negotiation.cc:1035] Waiting for connection context
0806 03:00:20.683287 (+ 460us) server_negotiation.cc:299] Negotiation successful
0806 03:00:20.683739 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":734,"thread_start_us":532,"threads_started":1}
I20260806 03:00:20.689039 10235 raft_consensus.cc:348] T 00000000000000000000000000000000 P 97ed33cf1323474e9dda4620b71416df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97ed33cf1323474e9dda4620b71416df" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 45429 } }
I20260806 03:00:20.689487 10235 raft_consensus.cc:374] T 00000000000000000000000000000000 P 97ed33cf1323474e9dda4620b71416df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:00:20.689613 10235 raft_consensus.cc:729] T 00000000000000000000000000000000 P 97ed33cf1323474e9dda4620b71416df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97ed33cf1323474e9dda4620b71416df, State: Initialized, Role: FOLLOWER
I20260806 03:00:20.690231 10235 consensus_queue.cc:260] T 00000000000000000000000000000000 P 97ed33cf1323474e9dda4620b71416df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97ed33cf1323474e9dda4620b71416df" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 45429 } }
I20260806 03:00:20.690541 10235 raft_consensus.cc:388] T 00000000000000000000000000000000 P 97ed33cf1323474e9dda4620b71416df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:00:20.690778 10235 raft_consensus.cc:482] T 00000000000000000000000000000000 P 97ed33cf1323474e9dda4620b71416df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:00:20.690969 10235 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 97ed33cf1323474e9dda4620b71416df [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:00:20.694065 10235 raft_consensus.cc:504] T 00000000000000000000000000000000 P 97ed33cf1323474e9dda4620b71416df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97ed33cf1323474e9dda4620b71416df" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 45429 } }
I20260806 03:00:20.694700 10235 leader_election.cc:302] T 00000000000000000000000000000000 P 97ed33cf1323474e9dda4620b71416df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 97ed33cf1323474e9dda4620b71416df; no voters:
I20260806 03:00:20.695374 10235 leader_election.cc:288] T 00000000000000000000000000000000 P 97ed33cf1323474e9dda4620b71416df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:00:20.695539 10240 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 97ed33cf1323474e9dda4620b71416df [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:00:20.696327 10240 raft_consensus.cc:686] T 00000000000000000000000000000000 P 97ed33cf1323474e9dda4620b71416df [term 1 LEADER]: Becoming Leader. State: Replica: 97ed33cf1323474e9dda4620b71416df, State: Running, Role: LEADER
I20260806 03:00:20.697227 10240 consensus_queue.cc:237] T 00000000000000000000000000000000 P 97ed33cf1323474e9dda4620b71416df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97ed33cf1323474e9dda4620b71416df" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 45429 } }
I20260806 03:00:20.697863 10235 sys_catalog.cc:564] T 00000000000000000000000000000000 P 97ed33cf1323474e9dda4620b71416df [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:00:20.700482 10242 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97ed33cf1323474e9dda4620b71416df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 97ed33cf1323474e9dda4620b71416df. Latest consensus state: current_term: 1 leader_uuid: "97ed33cf1323474e9dda4620b71416df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97ed33cf1323474e9dda4620b71416df" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 45429 } } }
I20260806 03:00:20.701143 10242 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97ed33cf1323474e9dda4620b71416df [sys.catalog]: This master's current role is: LEADER
I20260806 03:00:20.700685 10241 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97ed33cf1323474e9dda4620b71416df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "97ed33cf1323474e9dda4620b71416df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97ed33cf1323474e9dda4620b71416df" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 45429 } } }
I20260806 03:00:20.701797 10241 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97ed33cf1323474e9dda4620b71416df [sys.catalog]: This master's current role is: LEADER
I20260806 03:00:20.703222 10250 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:00:20.708600 10250 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:00:20.717828 10250 catalog_manager.cc:1347] Generated new cluster ID: 5c7335fe3279421aaf80cc7ca299aac6
I20260806 03:00:20.717980 10250 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:00:20.732414 10250 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:00:20.733851 10250 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:00:20.743780 10250 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 97ed33cf1323474e9dda4620b71416df: Generated new TSK 0
I20260806 03:00:20.745155 10250 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.74.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.193.keytab.
Entry for principal kudu/127.31.74.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.193.keytab.
I20260806 03:00:20.837457 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.193:0
--local_ip_for_outbound_sockets=127.31.74.193
--webserver_interface=127.31.74.193
--webserver_port=0
--tserver_master_addrs=127.31.74.254:45429
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.193.keytab
--principal=kudu/127.31.74.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:44985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:00:21.233256 10263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:00:21.233649 10263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:00:21.233855 10263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:00:21.242424 10263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:00:21.242636 10263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:00:21.242836 10263 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.193
I20260806 03:00:21.253331 10263 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:44985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.193.keytab
--principal=kudu/127.31.74.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.74.193
--webserver_port=0
--tserver_master_addrs=127.31.74.254:45429
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.193
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:00:21.255478 10263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:00:21.257428 10263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:00:21.273072 10269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:00:21.274145 10270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:00:21.274960 10272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:00:21.276259 10263 server_base.cc:1034] running on GCE node
I20260806 03:00:21.277318 10263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:00:21.278493 10263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:00:21.279835 10263 hybrid_clock.cc:648] HybridClock initialized: now 1785985221279746 us; error 80 us; skew 500 ppm
I20260806 03:00:21.280221 10263 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 03:00:21 dist-test-slave-dhph krb5kdc[10155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985221, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:00:21.289207 10263 init.cc:377] Logged in from keytab as kudu/127.31.74.193@KRBTEST.COM (short username kudu)
I20260806 03:00:21.291128 10263 webserver.cc:460] Webserver started at http://127.31.74.193:46083/ using document root <none> and password file <none>
I20260806 03:00:21.291967 10263 fs_manager.cc:359] Metadata directory not provided
I20260806 03:00:21.292114 10263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:00:21.292469 10263 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:00:21.296361 10263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/minicluster-data/ts-0/data/instance:
uuid: "d984066c5cbd4342b1c8b601a9ab7a25"
format_stamp: "Formatted at 2026-08-06 03:00:21 on dist-test-slave-dhph"
server_key: "67bda46b08e8f047ab62cfd97bdafcb4"
server_key_iv: "a627aabdcc9d9e8b038fb0a4093aa430"
server_key_version: "encryptionkey@0"
I20260806 03:00:21.297353 10263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/minicluster-data/ts-0/wal/instance:
uuid: "d984066c5cbd4342b1c8b601a9ab7a25"
format_stamp: "Formatted at 2026-08-06 03:00:21 on dist-test-slave-dhph"
server_key: "67bda46b08e8f047ab62cfd97bdafcb4"
server_key_iv: "a627aabdcc9d9e8b038fb0a4093aa430"
server_key_version: "encryptionkey@0"
I20260806 03:00:21.304948 10263 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.003s
I20260806 03:00:21.308861 10280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:00:21.310410 10263 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 03:00:21.310565 10263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/minicluster-data/ts-0/wal
uuid: "d984066c5cbd4342b1c8b601a9ab7a25"
format_stamp: "Formatted at 2026-08-06 03:00:21 on dist-test-slave-dhph"
server_key: "67bda46b08e8f047ab62cfd97bdafcb4"
server_key_iv: "a627aabdcc9d9e8b038fb0a4093aa430"
server_key_version: "encryptionkey@0"
I20260806 03:00:21.310878 10263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:00:21.331799 10263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:00:21.337411 10263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:00:21.337836 10263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:00:21.339123 10263 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:00:21.340951 10263 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:00:21.341053 10263 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:00:21.341202 10263 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:00:21.341271 10263 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:00:21.383524 10263 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.193:39073
I20260806 03:00:21.383741 10396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.193:39073 every 8 connection(s)
I20260806 03:00:21.384960 10263 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
I20260806 03:00:21.386943 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 10263
I20260806 03:00:21.387166 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/minicluster-data/ts-0/wal/instance
I20260806 03:00:21.387822 32043 external_mini_cluster.cc:1428] Setting key 4d978e4122c2da6d8148e5f351f0d69e
Aug 06 03:00:21 dist-test-slave-dhph krb5kdc[10155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985221, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.193@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 03:00:21.410413 10399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:00:21.388746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.193:40669 (local address 127.31.74.254:45429)
0806 03:00:21.389276 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0806 03:00:21.389282 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 03:00:21.390855 (+ 1573us) server_negotiation.cc:407] Connection header received
0806 03:00:21.392763 (+ 1908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:00:21.392769 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:00:21.392864 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:00:21.392985 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:00:21.394559 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:21.395340 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:21.396932 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:21.397359 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:21.397498 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:21.401737 (+ 4239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:00:21.401752 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:00:21.401756 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:00:21.401794 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:00:21.405532 (+ 3738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:00:21.407077 (+ 1545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:00:21.407083 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:00:21.407086 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:00:21.407309 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:00:21.407982 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:00:21.407987 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:00:21.407989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:00:21.408329 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:00:21.408465 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0806 03:00:21.409594 (+ 1129us) server_negotiation.cc:299] Negotiation successful
0806 03:00:21.409848 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":282,"threads_started":1}
I20260806 03:00:21.412052 10400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:00:21.389546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:45429 (local address 127.31.74.193:40669)
0806 03:00:21.390705 (+ 1159us) negotiation.cc:107] Waiting for socket to connect
0806 03:00:21.390743 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0806 03:00:21.392482 (+ 1739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:00:21.393252 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:00:21.393262 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:00:21.393801 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:00:21.394372 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:21.394389 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:21.395501 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:21.395506 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:21.396778 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:21.396793 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:21.397602 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:21.397609 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:21.397849 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:21.398679 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:00:21.398715 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:00:21.401439 (+ 2724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:00:21.405887 (+ 4448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:00:21.405902 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:00:21.405925 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:00:21.406787 (+ 862us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:00:21.407466 (+ 679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:00:21.407475 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:00:21.407482 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:00:21.407814 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:00:21.408495 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:00:21.408506 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:00:21.409063 (+ 557us) client_negotiation.cc:769] Sending connection context
0806 03:00:21.409441 (+ 378us) client_negotiation.cc:240] Negotiation successful
0806 03:00:21.410167 (+ 726us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":844,"thread_start_us":582,"threads_started":1}
I20260806 03:00:21.414319 10397 heartbeater.cc:347] Connected to a master server at 127.31.74.254:45429
I20260806 03:00:21.414752 10397 heartbeater.cc:464] Registering TS with master...
I20260806 03:00:21.415644 10397 heartbeater.cc:511] Master 127.31.74.254:45429 requested a full tablet report, sending...
I20260806 03:00:21.418354 10200 ts_manager.cc:194] Registered new tserver with Master: d984066c5cbd4342b1c8b601a9ab7a25 (127.31.74.193:39073)
I20260806 03:00:21.420140 10200 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.74.193@KRBTEST.COM'} at 127.31.74.193:40669
WARNING: no policy specified for kudu/127.31.74.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.194.keytab.
Entry for principal kudu/127.31.74.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.194.keytab.
I20260806 03:00:21.471249 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.194:0
--local_ip_for_outbound_sockets=127.31.74.194
--webserver_interface=127.31.74.194
--webserver_port=0
--tserver_master_addrs=127.31.74.254:45429
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.194.keytab
--principal=kudu/127.31.74.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:44985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:00:21.850724 10403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:00:21.851039 10403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:00:21.851207 10403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:00:21.859993 10403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:00:21.860137 10403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:00:21.860325 10403 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.194
I20260806 03:00:21.871366 10403 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:44985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.194.keytab
--principal=kudu/127.31.74.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.74.194
--webserver_port=0
--tserver_master_addrs=127.31.74.254:45429
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.194
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:00:21.872828 10403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:00:21.874257 10403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:00:21.890090 10408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:00:21.890153 10412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:00:21.890368 10410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:00:21.891194 10403 server_base.cc:1034] running on GCE node
I20260806 03:00:21.891968 10403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:00:21.893060 10403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:00:21.894310 10403 hybrid_clock.cc:648] HybridClock initialized: now 1785985221894229 us; error 59 us; skew 500 ppm
I20260806 03:00:21.894819 10403 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 03:00:21 dist-test-slave-dhph krb5kdc[10155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985221, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:00:21.899415 10403 init.cc:377] Logged in from keytab as kudu/127.31.74.194@KRBTEST.COM (short username kudu)
I20260806 03:00:21.901404 10403 webserver.cc:460] Webserver started at http://127.31.74.194:37381/ using document root <none> and password file <none>
I20260806 03:00:21.902242 10403 fs_manager.cc:359] Metadata directory not provided
I20260806 03:00:21.902415 10403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:00:21.902859 10403 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:00:21.906427 10403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/minicluster-data/ts-1/data/instance:
uuid: "3f2b8a79b2e34204a352bbbf8a8f1008"
format_stamp: "Formatted at 2026-08-06 03:00:21 on dist-test-slave-dhph"
server_key: "c7dd39d7aea8f14a6360dd115d653c4f"
server_key_iv: "371ba60b5317747968880712b73f0a2e"
server_key_version: "encryptionkey@0"
I20260806 03:00:21.907397 10403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/minicluster-data/ts-1/wal/instance:
uuid: "3f2b8a79b2e34204a352bbbf8a8f1008"
format_stamp: "Formatted at 2026-08-06 03:00:21 on dist-test-slave-dhph"
server_key: "c7dd39d7aea8f14a6360dd115d653c4f"
server_key_iv: "371ba60b5317747968880712b73f0a2e"
server_key_version: "encryptionkey@0"
I20260806 03:00:21.913771 10403 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.001s
I20260806 03:00:21.918342 10419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:00:21.920344 10403 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 03:00:21.920596 10403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/minicluster-data/ts-1/wal
uuid: "3f2b8a79b2e34204a352bbbf8a8f1008"
format_stamp: "Formatted at 2026-08-06 03:00:21 on dist-test-slave-dhph"
server_key: "c7dd39d7aea8f14a6360dd115d653c4f"
server_key_iv: "371ba60b5317747968880712b73f0a2e"
server_key_version: "encryptionkey@0"
I20260806 03:00:21.920876 10403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:00:21.953950 10403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:00:21.958814 10403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:00:21.959163 10403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:00:21.960485 10403 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:00:21.962164 10403 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:00:21.962311 10403 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:00:21.962478 10403 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:00:21.962563 10403 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:00:22.002271 10403 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.194:43441
I20260806 03:00:22.002557 10533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.194:43441 every 8 connection(s)
I20260806 03:00:22.003789 10403 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
I20260806 03:00:22.009903 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 10403
I20260806 03:00:22.010177 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/minicluster-data/ts-1/wal/instance
I20260806 03:00:22.010732 32043 external_mini_cluster.cc:1428] Setting key edf713fd8482db60494af73b774f1665
Aug 06 03:00:22 dist-test-slave-dhph krb5kdc[10155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985221, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.194@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 03:00:22.028208 10537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:00:22.007151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.194:34979 (local address 127.31.74.254:45429)
0806 03:00:22.007691 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0806 03:00:22.007696 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 03:00:22.008626 (+ 930us) server_negotiation.cc:407] Connection header received
0806 03:00:22.010988 (+ 2362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:00:22.010994 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:00:22.011104 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:00:22.011280 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:00:22.013098 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:22.013897 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:22.015819 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:22.016311 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:22.016464 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:22.020924 (+ 4460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:00:22.020937 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:00:22.020942 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:00:22.020985 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:00:22.024318 (+ 3333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:00:22.025252 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:00:22.025257 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:00:22.025260 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:00:22.025402 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:00:22.025912 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:00:22.025917 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:00:22.025919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:00:22.026282 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:00:22.026467 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0806 03:00:22.027516 (+ 1049us) server_negotiation.cc:299] Negotiation successful
0806 03:00:22.027779 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":286,"threads_started":1}
I20260806 03:00:22.029274 10538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:00:22.007648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:45429 (local address 127.31.74.194:34979)
0806 03:00:22.008441 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0806 03:00:22.008489 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 03:00:22.010550 (+ 2061us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:00:22.011546 (+ 996us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:00:22.011555 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:00:22.012261 (+ 706us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:00:22.012916 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:22.012935 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:22.014079 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:22.014084 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:22.015603 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:22.015618 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:22.016592 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:22.016600 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:22.016872 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:22.017661 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:00:22.017687 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:00:22.020555 (+ 2868us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:00:22.024534 (+ 3979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:00:22.024545 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:00:22.024576 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:00:22.025086 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:00:22.025532 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:00:22.025537 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:00:22.025541 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:00:22.025767 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:00:22.026437 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:00:22.026447 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:00:22.026998 (+ 551us) client_negotiation.cc:769] Sending connection context
0806 03:00:22.027414 (+ 416us) client_negotiation.cc:240] Negotiation successful
0806 03:00:22.028020 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":324,"threads_started":1}
I20260806 03:00:22.030989 10534 heartbeater.cc:347] Connected to a master server at 127.31.74.254:45429
I20260806 03:00:22.031351 10534 heartbeater.cc:464] Registering TS with master...
I20260806 03:00:22.032197 10534 heartbeater.cc:511] Master 127.31.74.254:45429 requested a full tablet report, sending...
I20260806 03:00:22.034363 10200 ts_manager.cc:194] Registered new tserver with Master: 3f2b8a79b2e34204a352bbbf8a8f1008 (127.31.74.194:43441)
I20260806 03:00:22.035375 10200 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.74.194@KRBTEST.COM'} at 127.31.74.194:34979
WARNING: no policy specified for kudu/127.31.74.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.195.keytab.
Entry for principal kudu/127.31.74.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.195.keytab.
I20260806 03:00:22.088074 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.195:0
--local_ip_for_outbound_sockets=127.31.74.195
--webserver_interface=127.31.74.195
--webserver_port=0
--tserver_master_addrs=127.31.74.254:45429
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.195.keytab
--principal=kudu/127.31.74.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:44985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 03:00:22.425235 10397 heartbeater.cc:503] Master 127.31.74.254:45429 was elected leader, sending a full tablet report...
W20260806 03:00:22.485416 10541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:00:22.485720 10541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:00:22.485901 10541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:00:22.493796 10541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:00:22.493922 10541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:00:22.494102 10541 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.195
I20260806 03:00:22.507036 10541 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:44985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.195.keytab
--principal=kudu/127.31.74.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.74.195
--webserver_port=0
--tserver_master_addrs=127.31.74.254:45429
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.195
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:00:22.508441 10541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:00:22.509861 10541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:00:22.525538 10547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:00:22.525870 10549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:00:22.526805 10546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:00:22.528066 10541 server_base.cc:1034] running on GCE node
I20260806 03:00:22.529863 10541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:00:22.531762 10541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:00:22.533041 10541 hybrid_clock.cc:648] HybridClock initialized: now 1785985222532978 us; error 47 us; skew 500 ppm
I20260806 03:00:22.533473 10541 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 03:00:22 dist-test-slave-dhph krb5kdc[10155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985222, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:00:22.537673 10541 init.cc:377] Logged in from keytab as kudu/127.31.74.195@KRBTEST.COM (short username kudu)
I20260806 03:00:22.539609 10541 webserver.cc:460] Webserver started at http://127.31.74.195:39597/ using document root <none> and password file <none>
I20260806 03:00:22.540413 10541 fs_manager.cc:359] Metadata directory not provided
I20260806 03:00:22.540573 10541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:00:22.540917 10541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:00:22.545908 10541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/minicluster-data/ts-2/data/instance:
uuid: "5dd0fb7a43ef4dd5acf130274005217e"
format_stamp: "Formatted at 2026-08-06 03:00:22 on dist-test-slave-dhph"
server_key: "13f471ccedeb903789d2c262d65fa090"
server_key_iv: "597debb1e01f63dbf631bacd55ee11dc"
server_key_version: "encryptionkey@0"
I20260806 03:00:22.547152 10541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/minicluster-data/ts-2/wal/instance:
uuid: "5dd0fb7a43ef4dd5acf130274005217e"
format_stamp: "Formatted at 2026-08-06 03:00:22 on dist-test-slave-dhph"
server_key: "13f471ccedeb903789d2c262d65fa090"
server_key_iv: "597debb1e01f63dbf631bacd55ee11dc"
server_key_version: "encryptionkey@0"
I20260806 03:00:22.553233 10541 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260806 03:00:22.557585 10556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:00:22.560393 10541 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.001s
I20260806 03:00:22.560643 10541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/minicluster-data/ts-2/wal
uuid: "5dd0fb7a43ef4dd5acf130274005217e"
format_stamp: "Formatted at 2026-08-06 03:00:22 on dist-test-slave-dhph"
server_key: "13f471ccedeb903789d2c262d65fa090"
server_key_iv: "597debb1e01f63dbf631bacd55ee11dc"
server_key_version: "encryptionkey@0"
I20260806 03:00:22.560904 10541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:00:22.612114 10541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:00:22.617586 10541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:00:22.617962 10541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:00:22.619731 10541 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:00:22.621635 10541 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:00:22.621778 10541 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:00:22.621961 10541 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:00:22.622047 10541 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:00:22.672115 10541 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.195:42393
I20260806 03:00:22.672219 10671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.195:42393 every 8 connection(s)
I20260806 03:00:22.673911 10541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
I20260806 03:00:22.676980 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 10541
I20260806 03:00:22.677206 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/minicluster-data/ts-2/wal/instance
I20260806 03:00:22.677836 32043 external_mini_cluster.cc:1428] Setting key 39de5be6c7c1ba1da3f8e848fc758aba
Aug 06 03:00:22 dist-test-slave-dhph krb5kdc[10155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985222, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.195@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 03:00:22.705924 10674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:00:22.679852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.195:52915 (local address 127.31.74.254:45429)
0806 03:00:22.680527 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0806 03:00:22.680535 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 03:00:22.681707 (+ 1172us) server_negotiation.cc:407] Connection header received
0806 03:00:22.683969 (+ 2262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:00:22.683977 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:00:22.684085 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:00:22.684230 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:00:22.686261 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:22.687486 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:22.690500 (+ 3014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:22.691128 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:22.691276 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:22.696263 (+ 4987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:00:22.696279 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:00:22.696284 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:00:22.696330 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:00:22.700648 (+ 4318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:00:22.702189 (+ 1541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:00:22.702268 (+ 79us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:00:22.702279 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:00:22.702549 (+ 270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:00:22.703262 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:00:22.703268 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:00:22.703272 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:00:22.703729 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:00:22.703923 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0806 03:00:22.705203 (+ 1280us) server_negotiation.cc:299] Negotiation successful
0806 03:00:22.705479 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":313,"threads_started":1}
I20260806 03:00:22.707914 10675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:00:22.680576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:45429 (local address 127.31.74.195:52915)
0806 03:00:22.681510 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0806 03:00:22.681562 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 03:00:22.683670 (+ 2108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:00:22.684541 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:00:22.684550 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:00:22.685167 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:00:22.686031 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:22.686077 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:22.687922 (+ 1845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:22.687928 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:22.690305 (+ 2377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:22.690322 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:22.691447 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:22.691469 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:22.691852 (+ 383us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:22.692783 (+ 931us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:00:22.692840 (+ 57us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:00:22.695942 (+ 3102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:00:22.700924 (+ 4982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:00:22.700939 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:00:22.700968 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:00:22.701875 (+ 907us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:00:22.702750 (+ 875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:00:22.702758 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:00:22.702766 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:00:22.703106 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:00:22.703925 (+ 819us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:00:22.703937 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:00:22.704528 (+ 591us) client_negotiation.cc:769] Sending connection context
0806 03:00:22.704957 (+ 429us) client_negotiation.cc:240] Negotiation successful
0806 03:00:22.705710 (+ 753us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"spinlock_wait_cycles":12544,"thread_start_us":448,"threads_started":1}
I20260806 03:00:22.709883 10672 heartbeater.cc:347] Connected to a master server at 127.31.74.254:45429
I20260806 03:00:22.710253 10672 heartbeater.cc:464] Registering TS with master...
I20260806 03:00:22.711225 10672 heartbeater.cc:511] Master 127.31.74.254:45429 requested a full tablet report, sending...
I20260806 03:00:22.713047 10200 ts_manager.cc:194] Registered new tserver with Master: 5dd0fb7a43ef4dd5acf130274005217e (127.31.74.195:42393)
I20260806 03:00:22.713903 10200 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.74.195@KRBTEST.COM'} at 127.31.74.195:52915
I20260806 03:00:22.723027 32043 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 03:00:22.744518 10674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:00:22.729009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34360 (local address 127.31.74.254:45429)
0806 03:00:22.729376 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0806 03:00:22.729388 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 03:00:22.729785 (+ 397us) server_negotiation.cc:407] Connection header received
0806 03:00:22.729961 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:00:22.729966 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:00:22.730083 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:00:22.730192 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:00:22.732190 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:22.732945 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:22.734456 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:22.735157 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:22.735263 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:22.738108 (+ 2845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:00:22.738120 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:00:22.738124 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:00:22.738165 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:00:22.741320 (+ 3155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:00:22.742147 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:00:22.742152 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:00:22.742154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:00:22.742290 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:00:22.742949 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:00:22.742953 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:00:22.742955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:00:22.743359 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:00:22.743824 (+ 465us) server_negotiation.cc:1035] Waiting for connection context
0806 03:00:22.744089 (+ 265us) server_negotiation.cc:299] Negotiation successful
0806 03:00:22.744282 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231}
I20260806 03:00:22.750120 10200 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34360:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 03:00:22.759985 10200 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 03:00:22.801329 10686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:00:22.775908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.194:43441 (local address 127.0.0.1:36382)
0806 03:00:22.788377 (+ 12469us) negotiation.cc:107] Waiting for socket to connect
0806 03:00:22.788410 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0806 03:00:22.788656 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:00:22.790655 (+ 1999us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:00:22.790661 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:00:22.790701 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:00:22.791410 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:22.791429 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:22.793572 (+ 2143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:22.793579 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:22.797108 (+ 3529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:22.797128 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:22.799640 (+ 2512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:22.799648 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:22.800382 (+ 734us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:22.800460 (+ 78us) client_negotiation.cc:769] Sending connection context
0806 03:00:22.800601 (+ 141us) client_negotiation.cc:240] Negotiation successful
0806 03:00:22.800823 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12263,"mutex_wait_us":92,"thread_start_us":12910,"threads_started":1}
I20260806 03:00:22.801345 10687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:00:22.776045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36382 (local address 127.31.74.194:43441)
0806 03:00:22.789554 (+ 13509us) server_negotiation.cc:206] Beginning negotiation
0806 03:00:22.789764 (+ 210us) server_negotiation.cc:399] Waiting for connection header
0806 03:00:22.789817 (+ 53us) server_negotiation.cc:407] Connection header received
0806 03:00:22.789974 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:00:22.789983 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:00:22.790325 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:00:22.790664 (+ 339us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:00:22.791672 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:22.793409 (+ 1737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:22.797341 (+ 3932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:22.799081 (+ 1740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:22.799231 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:22.799470 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0806 03:00:22.800720 (+ 1250us) server_negotiation.cc:299] Negotiation successful
0806 03:00:22.800946 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13340,"thread_start_us":13200,"threads_started":1}
I20260806 03:00:22.802254 10685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:00:22.775161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.195:42393 (local address 127.0.0.1:60718)
0806 03:00:22.786971 (+ 11810us) negotiation.cc:107] Waiting for socket to connect
0806 03:00:22.787034 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0806 03:00:22.787602 (+ 568us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:00:22.788986 (+ 1384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:00:22.789000 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:00:22.789068 (+ 68us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:00:22.789758 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:22.789777 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:22.793235 (+ 3458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:22.793241 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:22.797119 (+ 3878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:22.797129 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:22.800494 (+ 3365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:22.800505 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:22.800718 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:22.800761 (+ 43us) client_negotiation.cc:769] Sending connection context
0806 03:00:22.800875 (+ 114us) client_negotiation.cc:240] Negotiation successful
0806 03:00:22.801030 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11272,"mutex_wait_us":106,"thread_start_us":10872,"threads_started":1}
I20260806 03:00:22.802311 10690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:00:22.790420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:39073 (local address 127.0.0.1:33626)
0806 03:00:22.791536 (+ 1116us) negotiation.cc:107] Waiting for socket to connect
0806 03:00:22.791566 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0806 03:00:22.791708 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:00:22.792721 (+ 1013us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:00:22.792726 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:00:22.792766 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:00:22.793223 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:22.793251 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:22.795537 (+ 2286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:22.795542 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:22.798168 (+ 2626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:22.798187 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:22.801406 (+ 3219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:22.801417 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:22.801599 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:22.801649 (+ 50us) client_negotiation.cc:769] Sending connection context
0806 03:00:22.801759 (+ 110us) client_negotiation.cc:240] Negotiation successful
0806 03:00:22.801905 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":952,"thread_start_us":699,"threads_started":1}
I20260806 03:00:22.802835 10689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:00:22.788288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33626 (local address 127.31.74.193:39073)
0806 03:00:22.790328 (+ 2040us) server_negotiation.cc:206] Beginning negotiation
0806 03:00:22.790338 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 03:00:22.791879 (+ 1541us) server_negotiation.cc:407] Connection header received
0806 03:00:22.792013 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:00:22.792022 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:00:22.792506 (+ 484us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:00:22.792755 (+ 249us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:00:22.793461 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:22.795349 (+ 1888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:22.798407 (+ 3058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:22.799908 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:22.800056 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:22.800285 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0806 03:00:22.802052 (+ 1767us) server_negotiation.cc:299] Negotiation successful
0806 03:00:22.802290 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1816,"thread_start_us":1623,"threads_started":1}
I20260806 03:00:22.802853 10688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:00:22.775930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60718 (local address 127.31.74.195:42393)
0806 03:00:22.786002 (+ 10072us) server_negotiation.cc:206] Beginning negotiation
0806 03:00:22.786014 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 03:00:22.787976 (+ 1962us) server_negotiation.cc:407] Connection header received
0806 03:00:22.788152 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:00:22.788162 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:00:22.788589 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:00:22.790071 (+ 1482us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:00:22.790699 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:22.793007 (+ 2308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:22.797396 (+ 4389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:22.799381 (+ 1985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:22.799644 (+ 263us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:22.799949 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0806 03:00:22.800949 (+ 1000us) server_negotiation.cc:299] Negotiation successful
0806 03:00:22.801142 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9846,"thread_start_us":9632,"threads_started":1}
I20260806 03:00:22.806198 10329 tablet_service.cc:1467] Processing CreateTablet for tablet b9213c6d5c2e497f9867aaf8245b4aae (DEFAULT_TABLE table=test-table [id=6bb2b927e41348ee8a0cc4ea7a244bf0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:00:22.805939 10468 tablet_service.cc:1467] Processing CreateTablet for tablet b9213c6d5c2e497f9867aaf8245b4aae (DEFAULT_TABLE table=test-table [id=6bb2b927e41348ee8a0cc4ea7a244bf0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:00:22.806510 10605 tablet_service.cc:1467] Processing CreateTablet for tablet b9213c6d5c2e497f9867aaf8245b4aae (DEFAULT_TABLE table=test-table [id=6bb2b927e41348ee8a0cc4ea7a244bf0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:00:22.807821 10329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9213c6d5c2e497f9867aaf8245b4aae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:00:22.807808 10468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9213c6d5c2e497f9867aaf8245b4aae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:00:22.808187 10605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9213c6d5c2e497f9867aaf8245b4aae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:00:22.827793 10691 tablet_bootstrap.cc:492] T b9213c6d5c2e497f9867aaf8245b4aae P d984066c5cbd4342b1c8b601a9ab7a25: Bootstrap starting.
I20260806 03:00:22.828204 10695 tablet_bootstrap.cc:492] T b9213c6d5c2e497f9867aaf8245b4aae P 5dd0fb7a43ef4dd5acf130274005217e: Bootstrap starting.
I20260806 03:00:22.830447 10694 tablet_bootstrap.cc:492] T b9213c6d5c2e497f9867aaf8245b4aae P 3f2b8a79b2e34204a352bbbf8a8f1008: Bootstrap starting.
I20260806 03:00:22.832433 10691 tablet_bootstrap.cc:654] T b9213c6d5c2e497f9867aaf8245b4aae P d984066c5cbd4342b1c8b601a9ab7a25: Neither blocks nor log segments found. Creating new log.
I20260806 03:00:22.832432 10695 tablet_bootstrap.cc:654] T b9213c6d5c2e497f9867aaf8245b4aae P 5dd0fb7a43ef4dd5acf130274005217e: Neither blocks nor log segments found. Creating new log.
I20260806 03:00:22.833390 10694 tablet_bootstrap.cc:654] T b9213c6d5c2e497f9867aaf8245b4aae P 3f2b8a79b2e34204a352bbbf8a8f1008: Neither blocks nor log segments found. Creating new log.
I20260806 03:00:22.833954 10691 log.cc:824] T b9213c6d5c2e497f9867aaf8245b4aae P d984066c5cbd4342b1c8b601a9ab7a25: Log is configured to *not* fsync() on all Append() calls
I20260806 03:00:22.834645 10694 log.cc:824] T b9213c6d5c2e497f9867aaf8245b4aae P 3f2b8a79b2e34204a352bbbf8a8f1008: Log is configured to *not* fsync() on all Append() calls
I20260806 03:00:22.834650 10695 log.cc:824] T b9213c6d5c2e497f9867aaf8245b4aae P 5dd0fb7a43ef4dd5acf130274005217e: Log is configured to *not* fsync() on all Append() calls
I20260806 03:00:22.838835 10695 tablet_bootstrap.cc:492] T b9213c6d5c2e497f9867aaf8245b4aae P 5dd0fb7a43ef4dd5acf130274005217e: No bootstrap required, opened a new log
I20260806 03:00:22.838865 10691 tablet_bootstrap.cc:492] T b9213c6d5c2e497f9867aaf8245b4aae P d984066c5cbd4342b1c8b601a9ab7a25: No bootstrap required, opened a new log
I20260806 03:00:22.839061 10694 tablet_bootstrap.cc:492] T b9213c6d5c2e497f9867aaf8245b4aae P 3f2b8a79b2e34204a352bbbf8a8f1008: No bootstrap required, opened a new log
I20260806 03:00:22.839188 10691 ts_tablet_manager.cc:1397] T b9213c6d5c2e497f9867aaf8245b4aae P d984066c5cbd4342b1c8b601a9ab7a25: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.003s
I20260806 03:00:22.839179 10695 ts_tablet_manager.cc:1397] T b9213c6d5c2e497f9867aaf8245b4aae P 5dd0fb7a43ef4dd5acf130274005217e: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260806 03:00:22.839502 10694 ts_tablet_manager.cc:1397] T b9213c6d5c2e497f9867aaf8245b4aae P 3f2b8a79b2e34204a352bbbf8a8f1008: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260806 03:00:22.850921 10695 raft_consensus.cc:348] T b9213c6d5c2e497f9867aaf8245b4aae P 5dd0fb7a43ef4dd5acf130274005217e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd0fb7a43ef4dd5acf130274005217e" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 42393 } } peers { permanent_uuid: "d984066c5cbd4342b1c8b601a9ab7a25" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 39073 } } peers { permanent_uuid: "3f2b8a79b2e34204a352bbbf8a8f1008" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 43441 } }
I20260806 03:00:22.851428 10695 raft_consensus.cc:374] T b9213c6d5c2e497f9867aaf8245b4aae P 5dd0fb7a43ef4dd5acf130274005217e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:00:22.851540 10695 raft_consensus.cc:729] T b9213c6d5c2e497f9867aaf8245b4aae P 5dd0fb7a43ef4dd5acf130274005217e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dd0fb7a43ef4dd5acf130274005217e, State: Initialized, Role: FOLLOWER
I20260806 03:00:22.852244 10695 consensus_queue.cc:260] T b9213c6d5c2e497f9867aaf8245b4aae P 5dd0fb7a43ef4dd5acf130274005217e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd0fb7a43ef4dd5acf130274005217e" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 42393 } } peers { permanent_uuid: "d984066c5cbd4342b1c8b601a9ab7a25" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 39073 } } peers { permanent_uuid: "3f2b8a79b2e34204a352bbbf8a8f1008" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 43441 } }
I20260806 03:00:22.852550 10694 raft_consensus.cc:348] T b9213c6d5c2e497f9867aaf8245b4aae P 3f2b8a79b2e34204a352bbbf8a8f1008 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd0fb7a43ef4dd5acf130274005217e" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 42393 } } peers { permanent_uuid: "d984066c5cbd4342b1c8b601a9ab7a25" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 39073 } } peers { permanent_uuid: "3f2b8a79b2e34204a352bbbf8a8f1008" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 43441 } }
I20260806 03:00:22.852550 10691 raft_consensus.cc:348] T b9213c6d5c2e497f9867aaf8245b4aae P d984066c5cbd4342b1c8b601a9ab7a25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd0fb7a43ef4dd5acf130274005217e" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 42393 } } peers { permanent_uuid: "d984066c5cbd4342b1c8b601a9ab7a25" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 39073 } } peers { permanent_uuid: "3f2b8a79b2e34204a352bbbf8a8f1008" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 43441 } }
I20260806 03:00:22.853078 10691 raft_consensus.cc:374] T b9213c6d5c2e497f9867aaf8245b4aae P d984066c5cbd4342b1c8b601a9ab7a25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:00:22.853078 10694 raft_consensus.cc:374] T b9213c6d5c2e497f9867aaf8245b4aae P 3f2b8a79b2e34204a352bbbf8a8f1008 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:00:22.853199 10691 raft_consensus.cc:729] T b9213c6d5c2e497f9867aaf8245b4aae P d984066c5cbd4342b1c8b601a9ab7a25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d984066c5cbd4342b1c8b601a9ab7a25, State: Initialized, Role: FOLLOWER
I20260806 03:00:22.853214 10694 raft_consensus.cc:729] T b9213c6d5c2e497f9867aaf8245b4aae P 3f2b8a79b2e34204a352bbbf8a8f1008 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f2b8a79b2e34204a352bbbf8a8f1008, State: Initialized, Role: FOLLOWER
I20260806 03:00:22.853679 10672 heartbeater.cc:503] Master 127.31.74.254:45429 was elected leader, sending a full tablet report...
I20260806 03:00:22.853937 10691 consensus_queue.cc:260] T b9213c6d5c2e497f9867aaf8245b4aae P d984066c5cbd4342b1c8b601a9ab7a25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd0fb7a43ef4dd5acf130274005217e" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 42393 } } peers { permanent_uuid: "d984066c5cbd4342b1c8b601a9ab7a25" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 39073 } } peers { permanent_uuid: "3f2b8a79b2e34204a352bbbf8a8f1008" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 43441 } }
I20260806 03:00:22.853917 10694 consensus_queue.cc:260] T b9213c6d5c2e497f9867aaf8245b4aae P 3f2b8a79b2e34204a352bbbf8a8f1008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd0fb7a43ef4dd5acf130274005217e" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 42393 } } peers { permanent_uuid: "d984066c5cbd4342b1c8b601a9ab7a25" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 39073 } } peers { permanent_uuid: "3f2b8a79b2e34204a352bbbf8a8f1008" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 43441 } }
I20260806 03:00:22.854419 10695 ts_tablet_manager.cc:1428] T b9213c6d5c2e497f9867aaf8245b4aae P 5dd0fb7a43ef4dd5acf130274005217e: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260806 03:00:22.856390 10691 ts_tablet_manager.cc:1428] T b9213c6d5c2e497f9867aaf8245b4aae P d984066c5cbd4342b1c8b601a9ab7a25: Time spent starting tablet: real 0.017s user 0.010s sys 0.006s
I20260806 03:00:22.856564 10694 ts_tablet_manager.cc:1428] T b9213c6d5c2e497f9867aaf8245b4aae P 3f2b8a79b2e34204a352bbbf8a8f1008: Time spent starting tablet: real 0.017s user 0.016s sys 0.000s
I20260806 03:00:22.857106 10534 heartbeater.cc:503] Master 127.31.74.254:45429 was elected leader, sending a full tablet report...
I20260806 03:00:22.872045 10699 raft_consensus.cc:482] T b9213c6d5c2e497f9867aaf8245b4aae P 5dd0fb7a43ef4dd5acf130274005217e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:00:22.872375 10699 raft_consensus.cc:504] T b9213c6d5c2e497f9867aaf8245b4aae P 5dd0fb7a43ef4dd5acf130274005217e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd0fb7a43ef4dd5acf130274005217e" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 42393 } } peers { permanent_uuid: "d984066c5cbd4342b1c8b601a9ab7a25" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 39073 } } peers { permanent_uuid: "3f2b8a79b2e34204a352bbbf8a8f1008" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 43441 } }
I20260806 03:00:22.874178 10699 leader_election.cc:288] T b9213c6d5c2e497f9867aaf8245b4aae P 5dd0fb7a43ef4dd5acf130274005217e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d984066c5cbd4342b1c8b601a9ab7a25 (127.31.74.193:39073), 3f2b8a79b2e34204a352bbbf8a8f1008 (127.31.74.194:43441)
I20260806 03:00:22.881253 10687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:00:22.874761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.195:34247 (local address 127.31.74.194:43441)
0806 03:00:22.874981 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0806 03:00:22.874987 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 03:00:22.875245 (+ 258us) server_negotiation.cc:407] Connection header received
0806 03:00:22.875512 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:00:22.875518 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:00:22.875601 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:00:22.875742 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:00:22.876228 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:22.877104 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:22.879501 (+ 2397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:22.880308 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:22.880478 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:22.880558 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 03:00:22.880855 (+ 297us) server_negotiation.cc:299] Negotiation successful
0806 03:00:22.880987 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 03:00:22.881259 10675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:00:22.874567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.194:43441 (local address 127.31.74.195:34247)
0806 03:00:22.874985 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0806 03:00:22.875007 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 03:00:22.875324 (+ 317us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:00:22.875834 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:00:22.875842 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:00:22.875869 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:00:22.876072 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:22.876081 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:22.877255 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:22.877261 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:22.879331 (+ 2070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:22.879345 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:22.880482 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:22.880487 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:22.880676 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:22.880701 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 03:00:22.880813 (+ 112us) client_negotiation.cc:240] Negotiation successful
0806 03:00:22.880965 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252}
I20260806 03:00:22.882992 10702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:00:22.874727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:39073 (local address 127.31.74.195:49367)
0806 03:00:22.875642 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0806 03:00:22.875656 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 03:00:22.875808 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:00:22.876619 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:00:22.876626 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:00:22.876669 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:00:22.877199 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:22.877215 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:22.878933 (+ 1718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:22.878938 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:22.880495 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:00:22.880508 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:00:22.882031 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:00:22.882040 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:00:22.882199 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:22.882224 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 03:00:22.882371 (+ 147us) client_negotiation.cc:240] Negotiation successful
0806 03:00:22.882548 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":821,"thread_start_us":553,"threads_started":1}
I20260806 03:00:22.883059 10488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9213c6d5c2e497f9867aaf8245b4aae" candidate_uuid: "5dd0fb7a43ef4dd5acf130274005217e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f2b8a79b2e34204a352bbbf8a8f1008" is_pre_election: true
I20260806 03:00:22.883702 10488 raft_consensus.cc:2454] T b9213c6d5c2e497f9867aaf8245b4aae P 3f2b8a79b2e34204a352bbbf8a8f1008 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dd0fb7a43ef4dd5acf130274005217e in term 0.
I20260806 03:00:22.883811 10689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:00:22.874880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.195:49367 (local address 127.31.74.193:39073)
0806 03:00:22.875158 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0806 03:00:22.875168 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 03:00:22.875754 (+ 586us) server_negotiation.cc:407] Connection header received
0806 03:00:22.876009 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:00:22.876014 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:00:22.876292 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:00:22.876441 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:00:22.877435 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:22.878740 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:22.880687 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:22.881695 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:22.881893 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:22.882020 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0806 03:00:22.883031 (+ 1011us) server_negotiation.cc:299] Negotiation successful
0806 03:00:22.883248 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260806 03:00:22.884694 10561 leader_election.cc:302] T b9213c6d5c2e497f9867aaf8245b4aae P 5dd0fb7a43ef4dd5acf130274005217e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f2b8a79b2e34204a352bbbf8a8f1008, 5dd0fb7a43ef4dd5acf130274005217e; no voters:
I20260806 03:00:22.885232 10699 raft_consensus.cc:2781] T b9213c6d5c2e497f9867aaf8245b4aae P 5dd0fb7a43ef4dd5acf130274005217e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:00:22.885049 10349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9213c6d5c2e497f9867aaf8245b4aae" candidate_uuid: "5dd0fb7a43ef4dd5acf130274005217e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d984066c5cbd4342b1c8b601a9ab7a25" is_pre_election: true
I20260806 03:00:22.885421 10699 raft_consensus.cc:482] T b9213c6d5c2e497f9867aaf8245b4aae P 5dd0fb7a43ef4dd5acf130274005217e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:00:22.885546 10699 raft_consensus.cc:3037] T b9213c6d5c2e497f9867aaf8245b4aae P 5dd0fb7a43ef4dd5acf130274005217e [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:00:22.885857 10349 raft_consensus.cc:2454] T b9213c6d5c2e497f9867aaf8245b4aae P d984066c5cbd4342b1c8b601a9ab7a25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dd0fb7a43ef4dd5acf130274005217e in term 0.
W20260806 03:00:22.888463 10398 tablet.cc:2369] T b9213c6d5c2e497f9867aaf8245b4aae P d984066c5cbd4342b1c8b601a9ab7a25: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 03:00:22.888932 10699 raft_consensus.cc:504] T b9213c6d5c2e497f9867aaf8245b4aae P 5dd0fb7a43ef4dd5acf130274005217e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd0fb7a43ef4dd5acf130274005217e" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 42393 } } peers { permanent_uuid: "d984066c5cbd4342b1c8b601a9ab7a25" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 39073 } } peers { permanent_uuid: "3f2b8a79b2e34204a352bbbf8a8f1008" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 43441 } }
I20260806 03:00:22.889849 10699 leader_election.cc:288] T b9213c6d5c2e497f9867aaf8245b4aae P 5dd0fb7a43ef4dd5acf130274005217e [CANDIDATE]: Term 1 election: Requested vote from peers d984066c5cbd4342b1c8b601a9ab7a25 (127.31.74.193:39073), 3f2b8a79b2e34204a352bbbf8a8f1008 (127.31.74.194:43441)
I20260806 03:00:22.890343 10349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9213c6d5c2e497f9867aaf8245b4aae" candidate_uuid: "5dd0fb7a43ef4dd5acf130274005217e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d984066c5cbd4342b1c8b601a9ab7a25"
I20260806 03:00:22.890579 10349 raft_consensus.cc:3037] T b9213c6d5c2e497f9867aaf8245b4aae P d984066c5cbd4342b1c8b601a9ab7a25 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:00:22.890770 10488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9213c6d5c2e497f9867aaf8245b4aae" candidate_uuid: "5dd0fb7a43ef4dd5acf130274005217e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f2b8a79b2e34204a352bbbf8a8f1008"
I20260806 03:00:22.891177 10488 raft_consensus.cc:3037] T b9213c6d5c2e497f9867aaf8245b4aae P 3f2b8a79b2e34204a352bbbf8a8f1008 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:00:22.893783 10349 raft_consensus.cc:2454] T b9213c6d5c2e497f9867aaf8245b4aae P d984066c5cbd4342b1c8b601a9ab7a25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dd0fb7a43ef4dd5acf130274005217e in term 1.
I20260806 03:00:22.894203 10488 raft_consensus.cc:2454] T b9213c6d5c2e497f9867aaf8245b4aae P 3f2b8a79b2e34204a352bbbf8a8f1008 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dd0fb7a43ef4dd5acf130274005217e in term 1.
I20260806 03:00:22.894410 10558 leader_election.cc:302] T b9213c6d5c2e497f9867aaf8245b4aae P 5dd0fb7a43ef4dd5acf130274005217e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dd0fb7a43ef4dd5acf130274005217e, d984066c5cbd4342b1c8b601a9ab7a25; no voters:
I20260806 03:00:22.894832 10699 raft_consensus.cc:2781] T b9213c6d5c2e497f9867aaf8245b4aae P 5dd0fb7a43ef4dd5acf130274005217e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:00:22.895344 10699 raft_consensus.cc:686] T b9213c6d5c2e497f9867aaf8245b4aae P 5dd0fb7a43ef4dd5acf130274005217e [term 1 LEADER]: Becoming Leader. State: Replica: 5dd0fb7a43ef4dd5acf130274005217e, State: Running, Role: LEADER
I20260806 03:00:22.895973 10699 consensus_queue.cc:237] T b9213c6d5c2e497f9867aaf8245b4aae P 5dd0fb7a43ef4dd5acf130274005217e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd0fb7a43ef4dd5acf130274005217e" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 42393 } } peers { permanent_uuid: "d984066c5cbd4342b1c8b601a9ab7a25" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 39073 } } peers { permanent_uuid: "3f2b8a79b2e34204a352bbbf8a8f1008" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 43441 } }
I20260806 03:00:22.902007 10200 catalog_manager.cc:5495] T b9213c6d5c2e497f9867aaf8245b4aae P 5dd0fb7a43ef4dd5acf130274005217e reported cstate change: term changed from 0 to 1, leader changed from <none> to 5dd0fb7a43ef4dd5acf130274005217e (127.31.74.195). New cstate: current_term: 1 leader_uuid: "5dd0fb7a43ef4dd5acf130274005217e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd0fb7a43ef4dd5acf130274005217e" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 42393 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d984066c5cbd4342b1c8b601a9ab7a25" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 39073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3f2b8a79b2e34204a352bbbf8a8f1008" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 43441 } health_report { overall_health: UNKNOWN } } }
W20260806 03:00:22.926395 10673 tablet.cc:2369] T b9213c6d5c2e497f9867aaf8245b4aae P 5dd0fb7a43ef4dd5acf130274005217e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 03:00:22.940178 10688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:00:22.933125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60726 (local address 127.31.74.195:42393)
0806 03:00:22.933402 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0806 03:00:22.933407 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 03:00:22.933448 (+ 41us) server_negotiation.cc:407] Connection header received
0806 03:00:22.933816 (+ 368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:00:22.933821 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:00:22.933908 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:00:22.934063 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 03:00:22.934924 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:22.935683 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:22.937275 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:22.937672 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:22.937902 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:22.938745 (+ 843us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 03:00:22.939342 (+ 597us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 03:00:22.939472 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0806 03:00:22.939764 (+ 292us) server_negotiation.cc:299] Negotiation successful
0806 03:00:22.939886 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260806 03:00:22.951913 10488 raft_consensus.cc:1261] T b9213c6d5c2e497f9867aaf8245b4aae P 3f2b8a79b2e34204a352bbbf8a8f1008 [term 1 FOLLOWER]: Refusing update from remote peer 5dd0fb7a43ef4dd5acf130274005217e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:00:22.952389 10349 raft_consensus.cc:1261] T b9213c6d5c2e497f9867aaf8245b4aae P d984066c5cbd4342b1c8b601a9ab7a25 [term 1 FOLLOWER]: Refusing update from remote peer 5dd0fb7a43ef4dd5acf130274005217e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:00:22.953327 10699 consensus_queue.cc:1035] T b9213c6d5c2e497f9867aaf8245b4aae P 5dd0fb7a43ef4dd5acf130274005217e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f2b8a79b2e34204a352bbbf8a8f1008" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 43441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:00:22.953926 10703 consensus_queue.cc:1035] T b9213c6d5c2e497f9867aaf8245b4aae P 5dd0fb7a43ef4dd5acf130274005217e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d984066c5cbd4342b1c8b601a9ab7a25" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 39073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:00:22.968746 10706 mvcc.cc:205] Tried to move back new op lower bound from 7315395473196019712 to 7315395472987291648. Current Snapshot: MvccSnapshot[applied={T|T < 7315395473196019712}]
I20260806 03:00:22.992301 10199 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34360:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 03:00:22.992698 10199 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34360:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 03:00:22.998242 10199 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 97ed33cf1323474e9dda4620b71416df: Sending DeleteTablet for 3 replicas of tablet b9213c6d5c2e497f9867aaf8245b4aae
I20260806 03:00:22.999704 10605 tablet_service.cc:1514] Processing DeleteTablet for tablet b9213c6d5c2e497f9867aaf8245b4aae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:00:22 UTC) from {username='kudu', principal='kudu/127.31.74.254@KRBTEST.COM'} at 127.0.0.1:60718
I20260806 03:00:22.999886 10468 tablet_service.cc:1514] Processing DeleteTablet for tablet b9213c6d5c2e497f9867aaf8245b4aae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:00:22 UTC) from {username='kudu', principal='kudu/127.31.74.254@KRBTEST.COM'} at 127.0.0.1:36382
I20260806 03:00:23.000552 10329 tablet_service.cc:1514] Processing DeleteTablet for tablet b9213c6d5c2e497f9867aaf8245b4aae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:00:22 UTC) from {username='kudu', principal='kudu/127.31.74.254@KRBTEST.COM'} at 127.0.0.1:33626
I20260806 03:00:23.002055 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 10263
I20260806 03:00:23.004518 10716 tablet_replica.cc:330] T b9213c6d5c2e497f9867aaf8245b4aae P 5dd0fb7a43ef4dd5acf130274005217e: stopping tablet replica
I20260806 03:00:23.005064 10716 raft_consensus.cc:2229] T b9213c6d5c2e497f9867aaf8245b4aae P 5dd0fb7a43ef4dd5acf130274005217e [term 1 LEADER]: Raft consensus shutting down.
I20260806 03:00:23.005883 10716 raft_consensus.cc:2258] T b9213c6d5c2e497f9867aaf8245b4aae P 5dd0fb7a43ef4dd5acf130274005217e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:00:23.013541 10716 ts_tablet_manager.cc:1905] T b9213c6d5c2e497f9867aaf8245b4aae P 5dd0fb7a43ef4dd5acf130274005217e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:00:23.020105 10718 tablet_replica.cc:330] T b9213c6d5c2e497f9867aaf8245b4aae P d984066c5cbd4342b1c8b601a9ab7a25: stopping tablet replica
I20260806 03:00:23.020807 10718 raft_consensus.cc:2229] T b9213c6d5c2e497f9867aaf8245b4aae P d984066c5cbd4342b1c8b601a9ab7a25 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:00:23.021456 10718 raft_consensus.cc:2258] T b9213c6d5c2e497f9867aaf8245b4aae P d984066c5cbd4342b1c8b601a9ab7a25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:00:23.027976 10716 ts_tablet_manager.cc:1918] T b9213c6d5c2e497f9867aaf8245b4aae P 5dd0fb7a43ef4dd5acf130274005217e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 03:00:23.029168 10717 tablet_replica.cc:330] T b9213c6d5c2e497f9867aaf8245b4aae P 3f2b8a79b2e34204a352bbbf8a8f1008: stopping tablet replica
I20260806 03:00:23.029670 10717 raft_consensus.cc:2229] T b9213c6d5c2e497f9867aaf8245b4aae P 3f2b8a79b2e34204a352bbbf8a8f1008 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:00:23.030512 10717 raft_consensus.cc:2258] T b9213c6d5c2e497f9867aaf8245b4aae P 3f2b8a79b2e34204a352bbbf8a8f1008 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:00:23.030690 10718 ts_tablet_manager.cc:1905] T b9213c6d5c2e497f9867aaf8245b4aae P d984066c5cbd4342b1c8b601a9ab7a25: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:00:23.031314 10716 log.cc:1196] T b9213c6d5c2e497f9867aaf8245b4aae P 5dd0fb7a43ef4dd5acf130274005217e: Deleting WAL directory at /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/minicluster-data/ts-2/wal/wals/b9213c6d5c2e497f9867aaf8245b4aae
I20260806 03:00:23.032032 10716 ts_tablet_manager.cc:1939] T b9213c6d5c2e497f9867aaf8245b4aae P 5dd0fb7a43ef4dd5acf130274005217e: Deleting consensus metadata
I20260806 03:00:23.034194 10184 catalog_manager.cc:4841] TS 5dd0fb7a43ef4dd5acf130274005217e (127.31.74.195:42393): tablet b9213c6d5c2e497f9867aaf8245b4aae (table test-table [id=6bb2b927e41348ee8a0cc4ea7a244bf0]) successfully deleted
I20260806 03:00:23.036129 10717 ts_tablet_manager.cc:1905] T b9213c6d5c2e497f9867aaf8245b4aae P 3f2b8a79b2e34204a352bbbf8a8f1008: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:00:23.039330 10718 ts_tablet_manager.cc:1918] T b9213c6d5c2e497f9867aaf8245b4aae P d984066c5cbd4342b1c8b601a9ab7a25: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 03:00:23.039527 10718 log.cc:1196] T b9213c6d5c2e497f9867aaf8245b4aae P d984066c5cbd4342b1c8b601a9ab7a25: Deleting WAL directory at /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/minicluster-data/ts-0/wal/wals/b9213c6d5c2e497f9867aaf8245b4aae
I20260806 03:00:23.040264 10718 ts_tablet_manager.cc:1939] T b9213c6d5c2e497f9867aaf8245b4aae P d984066c5cbd4342b1c8b601a9ab7a25: Deleting consensus metadata
Aug 06 03:00:23 dist-test-slave-dhph krb5kdc[10155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.74.193@KRBTEST.COM
I20260806 03:00:23.042479 10184 catalog_manager.cc:4841] TS d984066c5cbd4342b1c8b601a9ab7a25 (127.31.74.193:39073): tablet b9213c6d5c2e497f9867aaf8245b4aae (table test-table [id=6bb2b927e41348ee8a0cc4ea7a244bf0]) successfully deleted
I20260806 03:00:23.044309 10717 ts_tablet_manager.cc:1918] T b9213c6d5c2e497f9867aaf8245b4aae P 3f2b8a79b2e34204a352bbbf8a8f1008: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 03:00:23.044523 10717 log.cc:1196] T b9213c6d5c2e497f9867aaf8245b4aae P 3f2b8a79b2e34204a352bbbf8a8f1008: Deleting WAL directory at /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785985105255475-32043-0/minicluster-data/ts-1/wal/wals/b9213c6d5c2e497f9867aaf8245b4aae
I20260806 03:00:23.045310 10717 ts_tablet_manager.cc:1939] T b9213c6d5c2e497f9867aaf8245b4aae P 3f2b8a79b2e34204a352bbbf8a8f1008: Deleting consensus metadata
I20260806 03:00:23.047936 10187 catalog_manager.cc:4841] TS 3f2b8a79b2e34204a352bbbf8a8f1008 (127.31.74.194:43441): tablet b9213c6d5c2e497f9867aaf8245b4aae (table test-table [id=6bb2b927e41348ee8a0cc4ea7a244bf0]) successfully deleted
I20260806 03:00:23.052171 10689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:00:23.003618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33634 (local address 127.31.74.193:39073)
0806 03:00:23.003821 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0806 03:00:23.003829 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 03:00:23.028747 (+ 24918us) server_negotiation.cc:407] Connection header received
0806 03:00:23.029197 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:00:23.029203 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:00:23.029292 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:00:23.029382 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:00:23.031978 (+ 2596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:23.033284 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:23.035885 (+ 2601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:23.036695 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:23.036825 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:23.043616 (+ 6791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:00:23.043630 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:00:23.043649 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:00:23.043709 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:00:23.048355 (+ 4646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:00:23.049455 (+ 1100us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:00:23.049469 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:00:23.049476 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:00:23.049675 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:00:23.050155 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:00:23.050161 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:00:23.050165 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:00:23.050780 (+ 615us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:00:23.051125 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0806 03:00:23.051459 (+ 334us) server_negotiation.cc:299] Negotiation successful
0806 03:00:23.051730 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 03:00:23.053016 10391 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 03:00:23.158947 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 10263
I20260806 03:00:23.184676 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 10403
Aug 06 03:00:23 dist-test-slave-dhph krb5kdc[10155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.74.194@KRBTEST.COM
I20260806 03:00:23.205094 10687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:00:23.185922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36392 (local address 127.31.74.194:43441)
0806 03:00:23.186124 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0806 03:00:23.186130 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 03:00:23.186154 (+ 24us) server_negotiation.cc:407] Connection header received
0806 03:00:23.186227 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:00:23.186233 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:00:23.186321 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:00:23.186444 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:00:23.188626 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:23.189426 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:23.191023 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:23.191436 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:23.191586 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:23.196843 (+ 5257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:00:23.196877 (+ 34us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:00:23.196909 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:00:23.196981 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:00:23.201656 (+ 4675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:00:23.202668 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:00:23.202676 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:00:23.202678 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:00:23.202898 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:00:23.203360 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:00:23.203365 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:00:23.203369 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:00:23.203805 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:00:23.204132 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0806 03:00:23.204479 (+ 347us) server_negotiation.cc:299] Negotiation successful
0806 03:00:23.204737 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260806 03:00:23.206450 10529 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 03:00:23.310039 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 10403
I20260806 03:00:23.336890 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 10541
Aug 06 03:00:23 dist-test-slave-dhph krb5kdc[10155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.74.195@KRBTEST.COM
I20260806 03:00:23.364357 10688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:00:23.337846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60732 (local address 127.31.74.195:42393)
0806 03:00:23.338037 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0806 03:00:23.338042 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 03:00:23.338061 (+ 19us) server_negotiation.cc:407] Connection header received
0806 03:00:23.338126 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:00:23.338133 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:00:23.338258 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:00:23.338359 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:00:23.340319 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:23.341610 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:23.344630 (+ 3020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:00:23.345281 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:00:23.345406 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:00:23.353140 (+ 7734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:00:23.353162 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:00:23.353182 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:00:23.353250 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:00:23.360236 (+ 6986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:00:23.361389 (+ 1153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:00:23.361398 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:00:23.361400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:00:23.361600 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:00:23.362080 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:00:23.362084 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:00:23.362086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:00:23.362520 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:00:23.362921 (+ 401us) server_negotiation.cc:1035] Waiting for connection context
0806 03:00:23.363443 (+ 522us) server_negotiation.cc:299] Negotiation successful
0806 03:00:23.363779 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 03:00:23.365453 10667 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 03:00:23.502570 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 10541
I20260806 03:00:23.529970 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 10168
I20260806 03:00:23.531981 10230 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 03:00:23.650118 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 10168
2026-08-06T03:00:23Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (5116 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 03:00:23 dist-test-slave-dhph krb5kdc[10726](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 03:00:23 dist-test-slave-dhph krb5kdc[10726](info): set up 2 sockets
Aug 06 03:00:23 dist-test-slave-dhph krb5kdc[10726](info): commencing operation
krb5kdc: starting...
W20260806 03:00:24.902904 32043 mini_kdc.cc:121] Time spent starting KDC: real 1.199s user 0.002s sys 0.048s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/krb5kdc/test-user.keytab.
Aug 06 03:00:25 dist-test-slave-dhph krb5kdc[10726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T03:00:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:00:25Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.31.74.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.31.74.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.31.74.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/krb5kdc/rangeradmin_127.31.74.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.31.74.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/krb5kdc/rangeradmin_127.31.74.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.31.74.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.31.74.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.31.74.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/krb5kdc/rangerlookup_127.31.74.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.31.74.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/krb5kdc/rangerlookup_127.31.74.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.31.74.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.31.74.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.31.74.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/krb5kdc/HTTP_127.31.74.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.31.74.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/krb5kdc/HTTP_127.31.74.212@KRBTEST.COM.keytab.
I20260806 03:00:25.428654 32043 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskGJ8w1x/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/postgres -l logfile start
2026-08-06 03:00:28.994 UTC [10756] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-06 03:00:28.994 UTC [10756] LOG: listening on IPv4 address "127.31.74.212", port 43291
2026-08-06 03:00:28.999 UTC [10756] LOG: listening on Unix socket "/tmp/.s.PGSQL.43291"
2026-08-06 03:00:29.026 UTC [10758] LOG: database system was shut down at 2026-08-06 03:00:26 UTC
2026-08-06 03:00:29.031 UTC [10756] LOG: database system is ready to accept connections
I20260806 03:00:30.111654 32043 mini_postgres.cc:96] Postgres bound to 43291
2026-08-06 03:00:30.141 UTC [10768] FATAL: database "slave" does not exist
127.31.74.212:43291 - accepting connections
I20260806 03:00:30.142886 32043 mini_ranger.cc:161] Starting Ranger...
I20260806 03:00:30.186444 32043 mini_ranger.cc:84] Created miniranger Postgres user
I20260806 03:00:31.101706 32043 mini_ranger.cc:87] Created ranger Postgres database
I20260806 03:00:31.101918 32043 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/ranger-admin
2026-08-06 03:00:31,497 [I] DB FLAVOR :POSTGRES
2026-08-06 03:00:31,497 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 03:00:31,497 [I] Checking connection..
2026-08-06 03:00:31,498 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJ8w1x/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGJ8w1x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.74.212:43291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-06 03:00:31,843 [I] Checking connection passed.
2026-08-06 03:00:31,843 [I] --------- Verifying version history table ---------
2026-08-06 03:00:31,843 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJ8w1x/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGJ8w1x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.74.212:43291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 03:00:32,222 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 03:00:32,222 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJ8w1x/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGJ8w1x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.74.212:43291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 03:00:32,541 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 03:00:32,541 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-06 03:00:32,541 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJ8w1x/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGJ8w1x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.74.212:43291/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskGJ8w1x/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-06 03:00:32.860 UTC [10837] WARNING: there is no transaction in progress
2026-08-06 03:00:32,870 [I] create_dbversion_catalog.sql file imported successfully
2026-08-06 03:00:32,871 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJ8w1x/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGJ8w1x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.74.212:43291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 03:00:33,204 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-06 03:00:33,205 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 03:00:33,205 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJ8w1x/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGJ8w1x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.74.212:43291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-06 03:00:33,517 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJ8w1x/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGJ8w1x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.74.212:43291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-06 03:00:33,802 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJ8w1x/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGJ8w1x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.74.212:43291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-dhph.c.gcp-upstream.internal','N') ;"
2026-08-06 03:00:34,130 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJ8w1x/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGJ8w1x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.74.212:43291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 03:00:34,480 [I] Table x_portal_user does not exist in database ranger
2026-08-06 03:00:34,480 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJ8w1x/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGJ8w1x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.74.212:43291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 03:00:34,812 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-06 03:00:34,813 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJ8w1x/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGJ8w1x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.74.212:43291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 03:00:35,133 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJ8w1x/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGJ8w1x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.74.212:43291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 03:00:35,467 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-06 03:00:35,467 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJ8w1x/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGJ8w1x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.74.212:43291/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskGJ8w1x/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-06 03:00:36.905 UTC [10949] WARNING: there is no transaction in progress
2026-08-06 03:00:36.937 UTC [10949] WARNING: there is no transaction in progress
2026-08-06 03:00:36.966 UTC [10949] WARNING: there is no transaction in progress
2026-08-06 03:00:36.996 UTC [10949] WARNING: there is no transaction in progress
2026-08-06 03:00:37.027 UTC [10949] WARNING: there is no transaction in progress
2026-08-06 03:00:37.061 UTC [10949] WARNING: there is no transaction in progress
2026-08-06 03:00:37.149 UTC [10949] WARNING: there is no transaction in progress
2026-08-06 03:00:37.161 UTC [10949] WARNING: there is no transaction in progress
2026-08-06 03:00:37.184 UTC [10949] WARNING: there is no transaction in progress
2026-08-06 03:00:37.200 UTC [10949] WARNING: there is no transaction in progress
2026-08-06 03:00:37.217 UTC [10949] WARNING: there is no transaction in progress
2026-08-06 03:00:37.238 UTC [10949] WARNING: there is no transaction in progress
2026-08-06 03:00:37.254 UTC [10949] WARNING: there is no transaction in progress
2026-08-06 03:00:37.267 UTC [10949] WARNING: there is no transaction in progress
2026-08-06 03:00:37.277 UTC [10949] WARNING: there is no transaction in progress
2026-08-06 03:00:37.287 UTC [10949] WARNING: there is no transaction in progress
2026-08-06 03:00:38,378 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-06 03:00:38,378 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJ8w1x/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGJ8w1x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.74.212:43291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-06 03:00:38,698 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-06 03:00:38,699 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJ8w1x/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGJ8w1x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.74.212:43291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 03:00:39,006 [I] Table x_portal_user already exists in database 'ranger'
2026-08-06 03:00:39,006 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJ8w1x/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGJ8w1x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.74.212:43291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 03:00:39,303 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-06 03:00:39,303 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJ8w1x/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGJ8w1x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.74.212:43291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 03:00:39,619 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJ8w1x/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGJ8w1x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.74.212:43291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 03:00:39,907 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJ8w1x/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGJ8w1x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.74.212:43291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-dhph.c.gcp-upstream.internal';"
2026-08-06 03:00:40,191 [I] CORE_DB_SCHEMA import status has been updated
2026-08-06 03:00:40,191 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJ8w1x/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGJ8w1x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.74.212:43291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-06 03:00:40,481 [I] DB_PATCHES have already been applied
I20260806 03:00:40.488232 32043 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/ranger-admin:/tmp/dist-test-taskGJ8w1x/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskGJ8w1x/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskGJ8w1x/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGJ8w1x/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260806 03:00:40.488337 32043 mini_ranger.cc:193] Using host: 127.31.74.212
I20260806 03:00:40.513914 32043 mini_ranger.cc:227] Ranger admin URL: http://127.31.74.212:54779
Aug 06, 2026 3:00:41 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 3:00:41 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 3:00:41 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/ranger-admin/ews/webapp
Aug 06, 2026 3:00:41 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 06, 2026 3:00:41 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 06, 2026 3:00:41 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/ranger-admin/ews/webapp].
Aug 06, 2026 3:00:41 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260806 03:00:41.915627 32043 mini_ranger.cc:160] Time spent starting Ranger: real 11.773s user 0.000s sys 0.089s
Aug 06, 2026 3:01:10 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 06, 2026 3:01:10 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 06, 2026 3:01:10 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 06, 2026 3:01:10 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 06, 2026 3:01:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 06, 2026 3:01:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 06, 2026 3:01:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 06, 2026 3:01:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 06, 2026 3:01:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 06, 2026 3:01:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 06, 2026 3:01:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 06, 2026 3:01:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 06, 2026 3:01:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 06, 2026 3:01:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 06, 2026 3:01:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 06, 2026 3:01:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 06, 2026 3:01:10 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 06, 2026 3:01:11 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260806 03:01:12.633736 32043 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.31.74.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.31.74.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.31.74.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/krb5kdc/rangerkms_127.31.74.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.31.74.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/krb5kdc/rangerkms_127.31.74.212@KRBTEST.COM.keytab.
W20260806 03:01:12.748417 32043 mini_kdc.cc:262] Time spent creating service keytab for rangerkms/127.31.74.212@KRBTEST.COM: real 0.113s user 0.003s sys 0.035s
WARNING: no policy specified for HTTP/127.31.74.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.31.74.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.31.74.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/krb5kdc/HTTP_127.31.74.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.31.74.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/krb5kdc/HTTP_127.31.74.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 06 03:01:12 dist-test-slave-dhph krb5kdc[10726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985272, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260806 03:01:12.977774 32043 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260806 03:01:13.022372 32043 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260806 03:01:14.744318 32043 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260806 03:01:14.744493 32043 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/ranger-kms
I20260806 03:01:14.744702 32043 mini_ranger_kms.cc:227] Using postgres at 127.31.74.212:43291
2026-08-06 03:01:15,518 [I] DB FLAVOR :POSTGRES
2026-08-06 03:01:15,519 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 03:01:15,519 [I] Checking connection
2026-08-06 03:01:15,519 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJ8w1x/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.74.212:43291/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-06 03:01:15,900 [I] connection success
2026-08-06 03:01:15,901 [I] --------- Verifying Ranger DB tables ---------
2026-08-06 03:01:15,901 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-06 03:01:15,901 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJ8w1x/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.74.212:43291/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-06 03:01:16,284 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-06 03:01:16,284 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 03:01:16,285 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-06 03:01:16,285 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJ8w1x/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.74.212:43291/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-06 03:01:16,720 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260806 03:01:17.597043 32043 mini_ranger_kms.cc:310] Created kms service
I20260806 03:01:17.933128 32043 mini_ranger_kms.cc:326] Created kudu user
I20260806 03:01:18.015620 32043 mini_ranger_kms.cc:343] Created rangerkms user
Aug 06 03:01:18 dist-test-slave-dhph krb5kdc[10726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985272, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.31.74.212@KRBTEST.COM
I20260806 03:01:18.585850 32043 mini_ranger_kms.cc:384] Added ranger policy
I20260806 03:01:18.588672 32043 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/ranger-kms:/tmp/dist-test-taskGJ8w1x/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGJ8w1x/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskGJ8w1x/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260806 03:01:18.588822 32043 mini_ranger_kms.cc:240] Using host: 127.31.74.212
I20260806 03:01:18.640508 32043 mini_ranger_kms.cc:276] Ranger KMS PID: 11141
I20260806 03:01:18.640777 32043 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.31.74.212:34185
Aug 06, 2026 3:01:20 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 3:01:20 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 3:01:20 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 06, 2026 3:01:20 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 06, 2026 3:01:20 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/ranger-kms/ews/webapp].
Aug 06, 2026 3:01:20 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260806 03:01:21.661535 32043 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 8.684s user 0.240s sys 0.613s
I20260806 03:01:21.661924 32043 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260806 03:01:21.662178 32043 mini_ranger_kms.cc:401] 127.31.74.212:34185/kms/v1/keys
Aug 06 03:01:21 dist-test-slave-dhph krb5kdc[10726](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1785985281, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.74.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 06 03:01:22 dist-test-slave-dhph krb5kdc[10726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985281, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.74.212@KRBTEST.COM for HTTP/127.31.74.212@KRBTEST.COM
Aug 06 03:01:23 dist-test-slave-dhph krb5kdc[10726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985281, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.74.212@KRBTEST.COM for HTTP/127.31.74.212@KRBTEST.COM
Aug 06 03:01:26 dist-test-slave-dhph krb5kdc[10726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.31.74.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab.
Entry for principal kudu/127.31.74.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab.
W20260806 03:01:26.496927 32043 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.31.74.254: real 0.106s user 0.000s sys 0.039s
I20260806 03:01:26.498865 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:33855
--webserver_interface=127.31.74.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab
--principal=kudu/127.31.74.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:33681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.74.254:33855
--ranger_config_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.74.212:34185/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:01:27.013046 11195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:01:27.013576 11195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:01:27.013734 11195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:01:27.030830 11195 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:01:27.031000 11195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:01:27.031054 11195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:01:27.031114 11195 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:01:27.031168 11195 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:01:27.044957 11195 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:33681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/master-0/wal
--ranger_kms_url=127.31.74.212:34185/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.31.74.254:33855
--ranger_config_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab
--principal=kudu/127.31.74.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:33855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.74.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:01:27.047856 11195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:01:27.049522 11195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:01:27.069265 11202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:01:27.069242 11201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:01:27.069528 11204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:01:27.071373 11195 server_base.cc:1034] running on GCE node
I20260806 03:01:27.072455 11195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:01:27.074643 11195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:01:27.075994 11195 hybrid_clock.cc:648] HybridClock initialized: now 1785985287075851 us; error 131 us; skew 500 ppm
I20260806 03:01:27.076642 11195 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 03:01:27 dist-test-slave-dhph krb5kdc[10726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985287, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:01:27.082190 11195 init.cc:377] Logged in from keytab as kudu/127.31.74.254@KRBTEST.COM (short username kudu)
I20260806 03:01:27.084604 11195 webserver.cc:460] Webserver started at http://127.31.74.254:39629/ using document root <none> and password file <none>
I20260806 03:01:27.085527 11195 fs_manager.cc:359] Metadata directory not provided
I20260806 03:01:27.085721 11195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:01:27.086100 11195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 03:01:27 dist-test-slave-dhph krb5kdc[10726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985287, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.254@KRBTEST.COM for HTTP/127.31.74.212@KRBTEST.COM
I20260806 03:01:27.198264 11195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/master-0/data/instance:
uuid: "8d55d44dee6c411f9dbc7d74f91bb168"
format_stamp: "Formatted at 2026-08-06 03:01:27 on dist-test-slave-dhph"
server_key: "6c3a7346d2622c26b7361ac1578fcb8b"
server_key_iv: "af5b317c8ef638450946d08ffa2f7ce7"
server_key_version: "kuduclusterkey@0"
I20260806 03:01:27.199605 11195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/master-0/wal/instance:
uuid: "8d55d44dee6c411f9dbc7d74f91bb168"
format_stamp: "Formatted at 2026-08-06 03:01:27 on dist-test-slave-dhph"
server_key: "6c3a7346d2622c26b7361ac1578fcb8b"
server_key_iv: "af5b317c8ef638450946d08ffa2f7ce7"
server_key_version: "kuduclusterkey@0"
I20260806 03:01:27.210384 11195 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.006s sys 0.001s
I20260806 03:01:27.249207 11211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:01:27.253072 11195 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.005s sys 0.001s
I20260806 03:01:27.253554 11195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/master-0/wal
uuid: "8d55d44dee6c411f9dbc7d74f91bb168"
format_stamp: "Formatted at 2026-08-06 03:01:27 on dist-test-slave-dhph"
server_key: "6c3a7346d2622c26b7361ac1578fcb8b"
server_key_iv: "af5b317c8ef638450946d08ffa2f7ce7"
server_key_version: "kuduclusterkey@0"
I20260806 03:01:27.254143 11195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:01:27.297799 11195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:01:27.303896 11195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:01:27.304306 11195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:01:27.345393 11195 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.254:33855
I20260806 03:01:27.345412 11263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.254:33855 every 8 connection(s)
I20260806 03:01:27.347440 11195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
I20260806 03:01:27.350497 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 11195
I20260806 03:01:27.350761 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/master-0/wal/instance
I20260806 03:01:27.351866 32043 external_mini_cluster.cc:1428] Setting key 4610596cf848060c9d1c30eb7da5e1a1
I20260806 03:01:27.359351 11264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:01:27.383172 11264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d55d44dee6c411f9dbc7d74f91bb168: Bootstrap starting.
Aug 06 03:01:27 dist-test-slave-dhph krb5kdc[10726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 03:01:27.389631 11264 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d55d44dee6c411f9dbc7d74f91bb168: Neither blocks nor log segments found. Creating new log.
I20260806 03:01:27.391556 11264 log.cc:824] T 00000000000000000000000000000000 P 8d55d44dee6c411f9dbc7d74f91bb168: Log is configured to *not* fsync() on all Append() calls
I20260806 03:01:27.399897 11264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d55d44dee6c411f9dbc7d74f91bb168: No bootstrap required, opened a new log
I20260806 03:01:27.404315 11267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:01:27.354685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34416 (local address 127.31.74.254:33855)
0806 03:01:27.364162 (+ 9477us) server_negotiation.cc:206] Beginning negotiation
0806 03:01:27.364192 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0806 03:01:27.364294 (+ 102us) server_negotiation.cc:407] Connection header received
0806 03:01:27.369433 (+ 5139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:01:27.369484 (+ 51us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:01:27.370216 (+ 732us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:01:27.370828 (+ 612us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:01:27.374147 (+ 3319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:01:27.376976 (+ 2829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:01:27.379650 (+ 2674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:01:27.381006 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:01:27.381417 (+ 411us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:01:27.387173 (+ 5756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:01:27.387186 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:01:27.387209 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:01:27.387280 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:01:27.396236 (+ 8956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:01:27.398262 (+ 2026us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:01:27.398276 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:01:27.398288 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:01:27.398541 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:01:27.399147 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:01:27.399153 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:01:27.399156 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:01:27.399967 (+ 811us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:01:27.401667 (+ 1700us) server_negotiation.cc:1035] Waiting for connection context
0806 03:01:27.402009 (+ 342us) server_negotiation.cc:299] Negotiation successful
0806 03:01:27.402572 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8322,"thread_start_us":8858,"threads_started":1}
I20260806 03:01:27.415624 11264 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d55d44dee6c411f9dbc7d74f91bb168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d55d44dee6c411f9dbc7d74f91bb168" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 33855 } }
I20260806 03:01:27.416975 11264 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d55d44dee6c411f9dbc7d74f91bb168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:01:27.417527 11264 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d55d44dee6c411f9dbc7d74f91bb168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d55d44dee6c411f9dbc7d74f91bb168, State: Initialized, Role: FOLLOWER
I20260806 03:01:27.418365 11264 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d55d44dee6c411f9dbc7d74f91bb168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d55d44dee6c411f9dbc7d74f91bb168" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 33855 } }
I20260806 03:01:27.418993 11264 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d55d44dee6c411f9dbc7d74f91bb168 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:01:27.419193 11264 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d55d44dee6c411f9dbc7d74f91bb168 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:01:27.419437 11264 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d55d44dee6c411f9dbc7d74f91bb168 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:01:27.422746 11264 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d55d44dee6c411f9dbc7d74f91bb168 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d55d44dee6c411f9dbc7d74f91bb168" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 33855 } }
I20260806 03:01:27.423461 11264 leader_election.cc:302] T 00000000000000000000000000000000 P 8d55d44dee6c411f9dbc7d74f91bb168 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d55d44dee6c411f9dbc7d74f91bb168; no voters:
I20260806 03:01:27.424291 11264 leader_election.cc:288] T 00000000000000000000000000000000 P 8d55d44dee6c411f9dbc7d74f91bb168 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:01:27.424446 11269 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d55d44dee6c411f9dbc7d74f91bb168 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:01:27.430441 11269 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d55d44dee6c411f9dbc7d74f91bb168 [term 1 LEADER]: Becoming Leader. State: Replica: 8d55d44dee6c411f9dbc7d74f91bb168, State: Running, Role: LEADER
I20260806 03:01:27.431939 11269 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d55d44dee6c411f9dbc7d74f91bb168 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d55d44dee6c411f9dbc7d74f91bb168" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 33855 } }
I20260806 03:01:27.433738 11264 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d55d44dee6c411f9dbc7d74f91bb168 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:01:27.436587 11271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d55d44dee6c411f9dbc7d74f91bb168 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d55d44dee6c411f9dbc7d74f91bb168. Latest consensus state: current_term: 1 leader_uuid: "8d55d44dee6c411f9dbc7d74f91bb168" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d55d44dee6c411f9dbc7d74f91bb168" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 33855 } } }
I20260806 03:01:27.437345 11271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d55d44dee6c411f9dbc7d74f91bb168 [sys.catalog]: This master's current role is: LEADER
I20260806 03:01:27.436837 11270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d55d44dee6c411f9dbc7d74f91bb168 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d55d44dee6c411f9dbc7d74f91bb168" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d55d44dee6c411f9dbc7d74f91bb168" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 33855 } } }
I20260806 03:01:27.437801 11270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d55d44dee6c411f9dbc7d74f91bb168 [sys.catalog]: This master's current role is: LEADER
I20260806 03:01:27.441721 11274 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:01:27.457526 11274 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:01:27.485884 11274 catalog_manager.cc:1347] Generated new cluster ID: 0b6de8c67434417191f54aaf34ec37de
I20260806 03:01:27.486138 11274 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:01:27.518039 11274 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:01:27.521441 11274 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:01:27.540155 11264 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260806 03:01:27.540649 11274 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d55d44dee6c411f9dbc7d74f91bb168: Generated new TSK 0
I20260806 03:01:27.542507 11274 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.74.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.193.keytab.
Entry for principal kudu/127.31.74.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.193.keytab.
I20260806 03:01:28.749428 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.74.212:34185/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.193:0
--local_ip_for_outbound_sockets=127.31.74.193
--webserver_interface=127.31.74.193
--webserver_port=0
--tserver_master_addrs=127.31.74.254:33855
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.193.keytab
--principal=kudu/127.31.74.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:33681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 03:01:29 dist-test-slave-dhph krb5kdc[10726](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1785985289, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260806 03:01:29.390053 11310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:01:29.390708 11310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:01:29.390923 11310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:01:29.404934 11310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:01:29.405293 11310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:01:29.405668 11310 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.193
I20260806 03:01:29.424340 11310 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:33681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.31.74.212:34185/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.193.keytab
--principal=kudu/127.31.74.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.74.193
--webserver_port=0
--tserver_master_addrs=127.31.74.254:33855
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.193
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:01:29.428012 11310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:01:29.430609 11310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:01:29.474673 11322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:01:29.476531 11320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:01:29.483603 11324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:01:29.484673 11310 server_base.cc:1034] running on GCE node
I20260806 03:01:29.486100 11310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:01:29.488368 11310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:01:29.489851 11310 hybrid_clock.cc:648] HybridClock initialized: now 1785985289489715 us; error 136 us; skew 500 ppm
I20260806 03:01:29.490418 11310 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 03:01:29 dist-test-slave-dhph krb5kdc[10726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985289, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:01:29.495848 11310 init.cc:377] Logged in from keytab as kudu/127.31.74.193@KRBTEST.COM (short username kudu)
I20260806 03:01:29.499012 11310 webserver.cc:460] Webserver started at http://127.31.74.193:45073/ using document root <none> and password file <none>
I20260806 03:01:29.500332 11310 fs_manager.cc:359] Metadata directory not provided
I20260806 03:01:29.500587 11310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:01:29.501108 11310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 03:01:29 dist-test-slave-dhph krb5kdc[10726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985289, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.193@KRBTEST.COM for HTTP/127.31.74.212@KRBTEST.COM
I20260806 03:01:29.533941 11310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/ts-0/data/instance:
uuid: "5467a76657c14a18abdfd10227bde360"
format_stamp: "Formatted at 2026-08-06 03:01:29 on dist-test-slave-dhph"
server_key: "ecc4702a2b8e3d3d44dbeae77c0a3bf7"
server_key_iv: "d262a54f849c46c767712f335d2835f2"
server_key_version: "kuduclusterkey@0"
I20260806 03:01:29.536875 11310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/ts-0/wal/instance:
uuid: "5467a76657c14a18abdfd10227bde360"
format_stamp: "Formatted at 2026-08-06 03:01:29 on dist-test-slave-dhph"
server_key: "ecc4702a2b8e3d3d44dbeae77c0a3bf7"
server_key_iv: "d262a54f849c46c767712f335d2835f2"
server_key_version: "kuduclusterkey@0"
I20260806 03:01:29.562484 11310 fs_manager.cc:691] Time spent creating directory manager: real 0.024s user 0.014s sys 0.007s
I20260806 03:01:29.590312 11331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:01:29.592895 11310 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260806 03:01:29.593171 11310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/ts-0/wal
uuid: "5467a76657c14a18abdfd10227bde360"
format_stamp: "Formatted at 2026-08-06 03:01:29 on dist-test-slave-dhph"
server_key: "ecc4702a2b8e3d3d44dbeae77c0a3bf7"
server_key_iv: "d262a54f849c46c767712f335d2835f2"
server_key_version: "kuduclusterkey@0"
I20260806 03:01:29.593490 11310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:01:29.620713 11310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:01:29.627868 11310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:01:29.628374 11310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:01:29.629959 11310 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:01:29.632445 11310 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:01:29.632647 11310 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260806 03:01:29.632910 11310 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:01:29.633062 11310 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:01:29.742265 11310 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.193:46607
I20260806 03:01:29.742350 11450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.193:46607 every 8 connection(s)
I20260806 03:01:29.744405 11310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
I20260806 03:01:29.751539 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 11310
I20260806 03:01:29.751856 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/ts-0/wal/instance
I20260806 03:01:29.753104 32043 external_mini_cluster.cc:1428] Setting key c6ee5a0001a417176ef1c0cd562011dd
Aug 06 03:01:29 dist-test-slave-dhph krb5kdc[10726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985289, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.193@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.74.194@KRBTEST.COM; defaulting to no policy
I20260806 03:01:29.816645 11453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:01:29.757608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:33855 (local address 127.31.74.193:48701)
0806 03:01:29.758927 (+ 1319us) negotiation.cc:107] Waiting for socket to connect
0806 03:01:29.758986 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0806 03:01:29.762516 (+ 3530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:01:29.776811 (+ 14295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:01:29.776831 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:01:29.777698 (+ 867us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:01:29.780044 (+ 2346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:01:29.780105 (+ 61us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:01:29.782899 (+ 2794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:01:29.782911 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:01:29.785794 (+ 2883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:01:29.785985 (+ 191us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:01:29.789838 (+ 3853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:01:29.789860 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:01:29.790297 (+ 437us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:01:29.791338 (+ 1041us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:01:29.791374 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:01:29.799756 (+ 8382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:01:29.808507 (+ 8751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:01:29.808521 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:01:29.808545 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:01:29.809402 (+ 857us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:01:29.810044 (+ 642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:01:29.810051 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:01:29.810055 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:01:29.810406 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:01:29.811419 (+ 1013us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:01:29.811430 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:01:29.812344 (+ 914us) client_negotiation.cc:769] Sending connection context
0806 03:01:29.814069 (+ 1725us) client_negotiation.cc:240] Negotiation successful
0806 03:01:29.814896 (+ 827us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":865,"thread_start_us":4983,"threads_started":1}
I20260806 03:01:29.817119 11454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:01:29.758496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.193:48701 (local address 127.31.74.254:33855)
0806 03:01:29.775688 (+ 17192us) server_negotiation.cc:206] Beginning negotiation
0806 03:01:29.775698 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 03:01:29.775732 (+ 34us) server_negotiation.cc:407] Connection header received
0806 03:01:29.775947 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:01:29.775957 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:01:29.776117 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:01:29.780454 (+ 4337us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:01:29.780771 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:01:29.782458 (+ 1687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:01:29.786317 (+ 3859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:01:29.787214 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:01:29.787382 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:01:29.800406 (+ 13024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:01:29.800417 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:01:29.800423 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:01:29.800481 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:01:29.807581 (+ 7100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:01:29.809632 (+ 2051us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:01:29.809640 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:01:29.809645 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:01:29.809854 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:01:29.810641 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:01:29.810649 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:01:29.810653 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:01:29.811135 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:01:29.811307 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0806 03:01:29.816352 (+ 5045us) server_negotiation.cc:299] Negotiation successful
0806 03:01:29.816678 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17020,"thread_start_us":16800,"threads_started":1}
I20260806 03:01:29.819937 11451 heartbeater.cc:347] Connected to a master server at 127.31.74.254:33855
I20260806 03:01:29.820923 11451 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.194@KRBTEST.COM" created.
I20260806 03:01:29.822991 11451 heartbeater.cc:511] Master 127.31.74.254:33855 requested a full tablet report, sending...
I20260806 03:01:29.828047 11229 ts_manager.cc:194] Registered new tserver with Master: 5467a76657c14a18abdfd10227bde360 (127.31.74.193:46607)
I20260806 03:01:29.830876 11229 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.74.193@KRBTEST.COM'} at 127.31.74.193:48701
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.194.keytab.
Entry for principal kudu/127.31.74.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.194.keytab.
W20260806 03:01:29.887054 32043 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.31.74.194: real 0.127s user 0.000s sys 0.036s
I20260806 03:01:29.888218 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.74.212:34185/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.194:0
--local_ip_for_outbound_sockets=127.31.74.194
--webserver_interface=127.31.74.194
--webserver_port=0
--tserver_master_addrs=127.31.74.254:33855
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.194.keytab
--principal=kudu/127.31.74.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:33681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 03:01:29 dist-test-slave-dhph krb5kdc[10726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985289, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.254@KRBTEST.COM for HTTP/127.31.74.212@KRBTEST.COM
Aug 06 03:01:30 dist-test-slave-dhph krb5kdc[10726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985289, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.254@KRBTEST.COM for HTTP/127.31.74.212@KRBTEST.COM
W20260806 03:01:30.559002 11457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:01:30.559589 11457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:01:30.559850 11457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:01:30.576615 11457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:01:30.577042 11457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:01:30.577304 11457 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.194
I20260806 03:01:30.598670 11457 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:33681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.31.74.212:34185/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.194.keytab
--principal=kudu/127.31.74.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.74.194
--webserver_port=0
--tserver_master_addrs=127.31.74.254:33855
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.194
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:01:30.601800 11457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:01:30.604280 11457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:01:30.649087 11465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:01:30.656579 11466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:01:30.668130 11457 server_base.cc:1034] running on GCE node
W20260806 03:01:30.673633 11468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:01:30.674983 11457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:01:30.683328 11457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:01:30.686916 11457 hybrid_clock.cc:648] HybridClock initialized: now 1785985290686736 us; error 129 us; skew 500 ppm
I20260806 03:01:30.687897 11457 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 03:01:30 dist-test-slave-dhph krb5kdc[10726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985290, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:01:30.695393 11457 init.cc:377] Logged in from keytab as kudu/127.31.74.194@KRBTEST.COM (short username kudu)
I20260806 03:01:30.726768 11457 webserver.cc:460] Webserver started at http://127.31.74.194:34237/ using document root <none> and password file <none>
I20260806 03:01:30.728094 11457 fs_manager.cc:359] Metadata directory not provided
I20260806 03:01:30.728366 11457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:01:30.728863 11457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 03:01:30 dist-test-slave-dhph krb5kdc[10726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985290, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.194@KRBTEST.COM for HTTP/127.31.74.212@KRBTEST.COM
I20260806 03:01:30.774025 11457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/ts-1/data/instance:
uuid: "c7e52d907a814506bfc514dd1d554398"
format_stamp: "Formatted at 2026-08-06 03:01:30 on dist-test-slave-dhph"
server_key: "e26d5cfa97cfd10f6c1c79191200293b"
server_key_iv: "3e6f918d5c81c8494253ecfe803dca1d"
server_key_version: "kuduclusterkey@0"
I20260806 03:01:30.780369 11457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/ts-1/wal/instance:
uuid: "c7e52d907a814506bfc514dd1d554398"
format_stamp: "Formatted at 2026-08-06 03:01:30 on dist-test-slave-dhph"
server_key: "e26d5cfa97cfd10f6c1c79191200293b"
server_key_iv: "3e6f918d5c81c8494253ecfe803dca1d"
server_key_version: "kuduclusterkey@0"
I20260806 03:01:30.796684 11457 fs_manager.cc:691] Time spent creating directory manager: real 0.015s user 0.009s sys 0.004s
I20260806 03:01:30.835983 11451 heartbeater.cc:503] Master 127.31.74.254:33855 was elected leader, sending a full tablet report...
I20260806 03:01:30.855675 11475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:01:30.859222 11457 fs_manager.cc:725] Time spent opening block manager: real 0.022s user 0.003s sys 0.016s
I20260806 03:01:30.859623 11457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/ts-1/wal
uuid: "c7e52d907a814506bfc514dd1d554398"
format_stamp: "Formatted at 2026-08-06 03:01:30 on dist-test-slave-dhph"
server_key: "e26d5cfa97cfd10f6c1c79191200293b"
server_key_iv: "3e6f918d5c81c8494253ecfe803dca1d"
server_key_version: "kuduclusterkey@0"
I20260806 03:01:30.860049 11457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:01:30.927429 11457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:01:30.936215 11457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:01:30.936882 11457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:01:30.945719 11457 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:01:30.953293 11457 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:01:30.953538 11457 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260806 03:01:30.953760 11457 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:01:30.953883 11457 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:01:31.080916 11602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.194:38489 every 8 connection(s)
I20260806 03:01:31.081386 11457 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.194:38489
I20260806 03:01:31.083477 11457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
I20260806 03:01:31.089357 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 11457
I20260806 03:01:31.089555 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/ts-1/wal/instance
I20260806 03:01:31.090075 32043 external_mini_cluster.cc:1428] Setting key c84776d0bde5fb2546365333382a0311
Aug 06 03:01:31 dist-test-slave-dhph krb5kdc[10726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985290, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.194@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 03:01:31.147243 11605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:01:31.088601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.194:59807 (local address 127.31.74.254:33855)
0806 03:01:31.106849 (+ 18248us) server_negotiation.cc:206] Beginning negotiation
0806 03:01:31.106859 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 03:01:31.106891 (+ 32us) server_negotiation.cc:407] Connection header received
0806 03:01:31.106983 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:01:31.106992 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:01:31.107113 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:01:31.107280 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:01:31.112335 (+ 5055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:01:31.113875 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:01:31.117380 (+ 3505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:01:31.119404 (+ 2024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:01:31.119647 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:01:31.131532 (+ 11885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:01:31.131539 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:01:31.131542 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:01:31.131686 (+ 144us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:01:31.136103 (+ 4417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:01:31.137538 (+ 1435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:01:31.137548 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:01:31.137554 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:01:31.137804 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:01:31.139349 (+ 1545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:01:31.139359 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:01:31.139366 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:01:31.140680 (+ 1314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:01:31.141600 (+ 920us) server_negotiation.cc:1035] Waiting for connection context
0806 03:01:31.144471 (+ 2871us) server_negotiation.cc:299] Negotiation successful
0806 03:01:31.144804 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18028,"thread_start_us":17443,"threads_started":1}
I20260806 03:01:31.151387 11606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:01:31.089494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:33855 (local address 127.31.74.194:59807)
0806 03:01:31.097562 (+ 8068us) negotiation.cc:107] Waiting for socket to connect
0806 03:01:31.097656 (+ 94us) client_negotiation.cc:174] Beginning negotiation
0806 03:01:31.105311 (+ 7655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:01:31.109575 (+ 4264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:01:31.109590 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:01:31.110393 (+ 803us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:01:31.111968 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:01:31.112015 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:01:31.114309 (+ 2294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:01:31.114318 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:01:31.117075 (+ 2757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:01:31.117119 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:01:31.120657 (+ 3538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:01:31.120668 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:01:31.121789 (+ 1121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:01:31.126183 (+ 4394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:01:31.126228 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:01:31.131059 (+ 4831us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:01:31.136417 (+ 5358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:01:31.136432 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:01:31.136458 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:01:31.137308 (+ 850us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:01:31.138325 (+ 1017us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:01:31.138334 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:01:31.138340 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:01:31.139125 (+ 785us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:01:31.142007 (+ 2882us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:01:31.142021 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:01:31.143094 (+ 1073us) client_negotiation.cc:769] Sending connection context
0806 03:01:31.148697 (+ 5603us) client_negotiation.cc:240] Negotiation successful
0806 03:01:31.149557 (+ 860us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7576,"thread_start_us":9294,"threads_started":1}
I20260806 03:01:31.153759 11603 heartbeater.cc:347] Connected to a master server at 127.31.74.254:33855
I20260806 03:01:31.154162 11603 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.31.74.195@KRBTEST.COM; defaulting to no policy
I20260806 03:01:31.155880 11603 heartbeater.cc:511] Master 127.31.74.254:33855 requested a full tablet report, sending...
I20260806 03:01:31.158089 11229 ts_manager.cc:194] Registered new tserver with Master: c7e52d907a814506bfc514dd1d554398 (127.31.74.194:38489)
I20260806 03:01:31.159317 11229 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.74.194@KRBTEST.COM'} at 127.31.74.194:59807
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.195.keytab.
Entry for principal kudu/127.31.74.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.195.keytab.
W20260806 03:01:31.215250 32043 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.31.74.195: real 0.122s user 0.001s sys 0.032s
I20260806 03:01:31.216939 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.74.212:34185/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.195:0
--local_ip_for_outbound_sockets=127.31.74.195
--webserver_interface=127.31.74.195
--webserver_port=0
--tserver_master_addrs=127.31.74.254:33855
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.195.keytab
--principal=kudu/127.31.74.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:33681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:01:31.729379 11609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:01:31.729705 11609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:01:31.729854 11609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:01:31.739142 11609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:01:31.739297 11609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:01:31.739478 11609 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.195
I20260806 03:01:31.751724 11609 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:33681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.31.74.212:34185/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.195.keytab
--principal=kudu/127.31.74.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.74.195
--webserver_port=0
--tserver_master_addrs=127.31.74.254:33855
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.195
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:01:31.753574 11609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:01:31.755167 11609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:01:31.770768 11614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:01:31.771580 11615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:01:31.773029 11618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:01:31.774041 11609 server_base.cc:1034] running on GCE node
I20260806 03:01:31.774876 11609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:01:31.775903 11609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:01:31.777200 11609 hybrid_clock.cc:648] HybridClock initialized: now 1785985291777179 us; error 60 us; skew 500 ppm
I20260806 03:01:31.777601 11609 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 03:01:31 dist-test-slave-dhph krb5kdc[10726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985291, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:01:31.781493 11609 init.cc:377] Logged in from keytab as kudu/127.31.74.195@KRBTEST.COM (short username kudu)
I20260806 03:01:31.783560 11609 webserver.cc:460] Webserver started at http://127.31.74.195:32875/ using document root <none> and password file <none>
I20260806 03:01:31.784394 11609 fs_manager.cc:359] Metadata directory not provided
I20260806 03:01:31.784555 11609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:01:31.784953 11609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 03:01:31 dist-test-slave-dhph krb5kdc[10726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985291, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.195@KRBTEST.COM for HTTP/127.31.74.212@KRBTEST.COM
I20260806 03:01:31.806821 11609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/ts-2/data/instance:
uuid: "1feaea8c263644cebeea7170f5ba31be"
format_stamp: "Formatted at 2026-08-06 03:01:31 on dist-test-slave-dhph"
server_key: "3fe7d0f5d109c48e0c762def967971da"
server_key_iv: "1dfdb7780b7ac3f0f7e83c6bf870083e"
server_key_version: "kuduclusterkey@0"
I20260806 03:01:31.807950 11609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/ts-2/wal/instance:
uuid: "1feaea8c263644cebeea7170f5ba31be"
format_stamp: "Formatted at 2026-08-06 03:01:31 on dist-test-slave-dhph"
server_key: "3fe7d0f5d109c48e0c762def967971da"
server_key_iv: "1dfdb7780b7ac3f0f7e83c6bf870083e"
server_key_version: "kuduclusterkey@0"
I20260806 03:01:31.814762 11609 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260806 03:01:31.836476 11625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:01:31.838312 11609 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260806 03:01:31.838537 11609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/ts-2/wal
uuid: "1feaea8c263644cebeea7170f5ba31be"
format_stamp: "Formatted at 2026-08-06 03:01:31 on dist-test-slave-dhph"
server_key: "3fe7d0f5d109c48e0c762def967971da"
server_key_iv: "1dfdb7780b7ac3f0f7e83c6bf870083e"
server_key_version: "kuduclusterkey@0"
I20260806 03:01:31.839207 11609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:01:31.877251 11609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:01:31.883019 11609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:01:31.883533 11609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:01:31.884934 11609 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:01:31.887491 11609 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:01:31.887616 11609 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:01:31.887805 11609 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:01:31.887876 11609 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:01:31.943761 11609 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.195:34281
I20260806 03:01:31.943940 11751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.195:34281 every 8 connection(s)
I20260806 03:01:31.945552 11609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
I20260806 03:01:31.955921 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 11609
I20260806 03:01:31.956276 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/ts-2/wal/instance
I20260806 03:01:31.956941 32043 external_mini_cluster.cc:1428] Setting key 15cdfadffb23eea4265c07c5bc535bf0
Aug 06 03:01:31 dist-test-slave-dhph krb5kdc[10726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985291, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.195@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 03:01:31.979547 11754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:01:31.949987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.195:37245 (local address 127.31.74.254:33855)
0806 03:01:31.950797 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0806 03:01:31.950807 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 03:01:31.953734 (+ 2927us) server_negotiation.cc:407] Connection header received
0806 03:01:31.956421 (+ 2687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:01:31.956428 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:01:31.956547 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:01:31.956703 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:01:31.959476 (+ 2773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:01:31.960915 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:01:31.963972 (+ 3057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:01:31.964479 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:01:31.964652 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:01:31.969358 (+ 4706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:01:31.969364 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:01:31.969369 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:01:31.969416 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:01:31.974308 (+ 4892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:01:31.976016 (+ 1708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:01:31.976022 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:01:31.976027 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:01:31.976296 (+ 269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:01:31.976939 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:01:31.976943 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:01:31.976945 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:01:31.977296 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:01:31.977452 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0806 03:01:31.978744 (+ 1292us) server_negotiation.cc:299] Negotiation successful
0806 03:01:31.979069 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":343,"threads_started":1}
I20260806 03:01:31.980068 11755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:01:31.951401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:33855 (local address 127.31.74.195:37245)
0806 03:01:31.953288 (+ 1887us) negotiation.cc:107] Waiting for socket to connect
0806 03:01:31.953379 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0806 03:01:31.955939 (+ 2560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:01:31.957171 (+ 1232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:01:31.957192 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:01:31.957896 (+ 704us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:01:31.959130 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:01:31.959208 (+ 78us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:01:31.961179 (+ 1971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:01:31.961185 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:01:31.963451 (+ 2266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:01:31.963481 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:01:31.964940 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:01:31.964950 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:01:31.965299 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:01:31.966078 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:01:31.966104 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:01:31.968971 (+ 2867us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:01:31.974893 (+ 5922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:01:31.974903 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:01:31.974921 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:01:31.975791 (+ 870us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:01:31.976535 (+ 744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:01:31.976539 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:01:31.976541 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:01:31.976759 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:01:31.977454 (+ 695us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:01:31.977462 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:01:31.977868 (+ 406us) client_negotiation.cc:769] Sending connection context
0806 03:01:31.978140 (+ 272us) client_negotiation.cc:240] Negotiation successful
0806 03:01:31.978763 (+ 623us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1347,"thread_start_us":963,"threads_started":1}
I20260806 03:01:31.982081 11752 heartbeater.cc:347] Connected to a master server at 127.31.74.254:33855
I20260806 03:01:31.982573 11752 heartbeater.cc:464] Registering TS with master...
I20260806 03:01:31.983860 11752 heartbeater.cc:511] Master 127.31.74.254:33855 requested a full tablet report, sending...
I20260806 03:01:31.986162 11229 ts_manager.cc:194] Registered new tserver with Master: 1feaea8c263644cebeea7170f5ba31be (127.31.74.195:34281)
I20260806 03:01:31.987761 11229 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.74.195@KRBTEST.COM'} at 127.31.74.195:37245
I20260806 03:01:31.990051 32043 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 03:01:32.014541 11754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:01:31.996020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34422 (local address 127.31.74.254:33855)
0806 03:01:31.996314 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0806 03:01:31.996322 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 03:01:31.996780 (+ 458us) server_negotiation.cc:407] Connection header received
0806 03:01:31.997016 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:01:31.997022 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:01:31.997124 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:01:31.997248 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:01:31.999344 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:01:32.000367 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:01:32.002505 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:01:32.003130 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:01:32.003310 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:01:32.007162 (+ 3852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:01:32.007168 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:01:32.007172 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:01:32.007227 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:01:32.011162 (+ 3935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:01:32.012181 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:01:32.012187 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:01:32.012190 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:01:32.012340 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:01:32.012789 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:01:32.012796 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:01:32.012800 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:01:32.013136 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:01:32.013424 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0806 03:01:32.013821 (+ 397us) server_negotiation.cc:299] Negotiation successful
0806 03:01:32.014148 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260806 03:01:32.020535 11229 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34422:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 03:01:32.030822 11229 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 03:01:32.073640 11765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:01:32.049415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.194:38489 (local address 127.0.0.1:34906)
0806 03:01:32.056881 (+ 7466us) negotiation.cc:107] Waiting for socket to connect
0806 03:01:32.056994 (+ 113us) client_negotiation.cc:174] Beginning negotiation
0806 03:01:32.057409 (+ 415us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:01:32.063132 (+ 5723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:01:32.063145 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:01:32.063181 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:01:32.063749 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:01:32.063762 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:01:32.066394 (+ 2632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:01:32.066400 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:01:32.069525 (+ 3125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:01:32.069569 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:01:32.072307 (+ 2738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:01:32.072315 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:01:32.072620 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:01:32.072723 (+ 103us) client_negotiation.cc:769] Sending connection context
0806 03:01:32.072862 (+ 139us) client_negotiation.cc:240] Negotiation successful
0806 03:01:32.073098 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6960,"mutex_wait_us":25,"thread_start_us":6637,"threads_started":1}
I20260806 03:01:32.074913 11766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:01:32.049618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34906 (local address 127.31.74.194:38489)
0806 03:01:32.050316 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0806 03:01:32.050326 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 03:01:32.059931 (+ 9605us) server_negotiation.cc:407] Connection header received
0806 03:01:32.060116 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:01:32.060130 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:01:32.062804 (+ 2674us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:01:32.063125 (+ 321us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:01:32.064382 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:01:32.066129 (+ 1747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:01:32.070000 (+ 3871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:01:32.072013 (+ 2013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:01:32.072222 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:01:32.072615 (+ 393us) server_negotiation.cc:1035] Waiting for connection context
0806 03:01:32.074165 (+ 1550us) server_negotiation.cc:299] Negotiation successful
0806 03:01:32.074370 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":361,"threads_started":1}
I20260806 03:01:32.076170 11768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:01:32.051306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:46607 (local address 127.0.0.1:36358)
0806 03:01:32.062810 (+ 11504us) negotiation.cc:107] Waiting for socket to connect
0806 03:01:32.062839 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0806 03:01:32.064270 (+ 1431us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:01:32.065850 (+ 1580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:01:32.065856 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:01:32.065885 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:01:32.067403 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:01:32.067498 (+ 95us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:01:32.070040 (+ 2542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:01:32.070352 (+ 312us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:01:32.072717 (+ 2365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:01:32.072732 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:01:32.075278 (+ 2546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:01:32.075284 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:01:32.075420 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:01:32.075445 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 03:01:32.075567 (+ 122us) client_negotiation.cc:240] Negotiation successful
0806 03:01:32.075711 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11308,"mutex_wait_us":31,"thread_start_us":7651,"threads_started":1}
I20260806 03:01:32.077155 11770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:01:32.052124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36358 (local address 127.31.74.193:46607)
0806 03:01:32.063985 (+ 11861us) server_negotiation.cc:206] Beginning negotiation
0806 03:01:32.063992 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 03:01:32.064020 (+ 28us) server_negotiation.cc:407] Connection header received
0806 03:01:32.064563 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:01:32.064573 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:01:32.065291 (+ 718us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:01:32.065541 (+ 250us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:01:32.067751 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:01:32.069770 (+ 2019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:01:32.072942 (+ 3172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:01:32.074550 (+ 1608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:01:32.074731 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:01:32.075018 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0806 03:01:32.076524 (+ 1506us) server_negotiation.cc:299] Negotiation successful
0806 03:01:32.076718 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11638,"thread_start_us":11416,"threads_started":1}
I20260806 03:01:32.079531 11531 tablet_service.cc:1467] Processing CreateTablet for tablet 79b79abb17b843b39436a408ff158fc0 (DEFAULT_TABLE table=test-table [id=5904dc936f3c4334bdd2cc760e081627]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:01:32.081269 11531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79b79abb17b843b39436a408ff158fc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:01:32.081303 11383 tablet_service.cc:1467] Processing CreateTablet for tablet 79b79abb17b843b39436a408ff158fc0 (DEFAULT_TABLE table=test-table [id=5904dc936f3c4334bdd2cc760e081627]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:01:32.082859 11383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79b79abb17b843b39436a408ff158fc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:01:32.088295 11769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:01:32.050929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.195:34281 (local address 127.0.0.1:41816)
0806 03:01:32.059403 (+ 8474us) negotiation.cc:107] Waiting for socket to connect
0806 03:01:32.059436 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0806 03:01:32.059664 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:01:32.066142 (+ 6478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:01:32.066156 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:01:32.066236 (+ 80us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:01:32.067544 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:01:32.067562 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:01:32.070057 (+ 2495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:01:32.070064 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:01:32.078250 (+ 8186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:01:32.078270 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:01:32.086980 (+ 8710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:01:32.086990 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:01:32.087262 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:01:32.087321 (+ 59us) client_negotiation.cc:769] Sending connection context
0806 03:01:32.087467 (+ 146us) client_negotiation.cc:240] Negotiation successful
0806 03:01:32.087648 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8282,"mutex_wait_us":1923,"thread_start_us":13098,"threads_started":1}
I20260806 03:01:32.088792 11767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:01:32.050365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41816 (local address 127.31.74.195:34281)
0806 03:01:32.065274 (+ 14909us) server_negotiation.cc:206] Beginning negotiation
0806 03:01:32.065291 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 03:01:32.065332 (+ 41us) server_negotiation.cc:407] Connection header received
0806 03:01:32.065429 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:01:32.065435 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:01:32.065666 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:01:32.066712 (+ 1046us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:01:32.067848 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:01:32.069774 (+ 1926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:01:32.078757 (+ 8983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:01:32.080523 (+ 1766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:01:32.080700 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:01:32.080980 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0806 03:01:32.087719 (+ 6739us) server_negotiation.cc:299] Negotiation successful
0806 03:01:32.088224 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14659,"thread_start_us":14484,"threads_started":1}
I20260806 03:01:32.095337 11678 tablet_service.cc:1467] Processing CreateTablet for tablet 79b79abb17b843b39436a408ff158fc0 (DEFAULT_TABLE table=test-table [id=5904dc936f3c4334bdd2cc760e081627]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:01:32.098237 11678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79b79abb17b843b39436a408ff158fc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:01:32.098381 11771 tablet_bootstrap.cc:492] T 79b79abb17b843b39436a408ff158fc0 P c7e52d907a814506bfc514dd1d554398: Bootstrap starting.
I20260806 03:01:32.104051 11771 tablet_bootstrap.cc:654] T 79b79abb17b843b39436a408ff158fc0 P c7e52d907a814506bfc514dd1d554398: Neither blocks nor log segments found. Creating new log.
I20260806 03:01:32.105353 11771 log.cc:824] T 79b79abb17b843b39436a408ff158fc0 P c7e52d907a814506bfc514dd1d554398: Log is configured to *not* fsync() on all Append() calls
I20260806 03:01:32.108206 11775 tablet_bootstrap.cc:492] T 79b79abb17b843b39436a408ff158fc0 P 5467a76657c14a18abdfd10227bde360: Bootstrap starting.
I20260806 03:01:32.109661 11771 tablet_bootstrap.cc:492] T 79b79abb17b843b39436a408ff158fc0 P c7e52d907a814506bfc514dd1d554398: No bootstrap required, opened a new log
I20260806 03:01:32.109987 11771 ts_tablet_manager.cc:1397] T 79b79abb17b843b39436a408ff158fc0 P c7e52d907a814506bfc514dd1d554398: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.004s
I20260806 03:01:32.115919 11775 tablet_bootstrap.cc:654] T 79b79abb17b843b39436a408ff158fc0 P 5467a76657c14a18abdfd10227bde360: Neither blocks nor log segments found. Creating new log.
I20260806 03:01:32.118266 11776 tablet_bootstrap.cc:492] T 79b79abb17b843b39436a408ff158fc0 P 1feaea8c263644cebeea7170f5ba31be: Bootstrap starting.
I20260806 03:01:32.118499 11775 log.cc:824] T 79b79abb17b843b39436a408ff158fc0 P 5467a76657c14a18abdfd10227bde360: Log is configured to *not* fsync() on all Append() calls
I20260806 03:01:32.124065 11775 tablet_bootstrap.cc:492] T 79b79abb17b843b39436a408ff158fc0 P 5467a76657c14a18abdfd10227bde360: No bootstrap required, opened a new log
I20260806 03:01:32.124469 11775 ts_tablet_manager.cc:1397] T 79b79abb17b843b39436a408ff158fc0 P 5467a76657c14a18abdfd10227bde360: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.003s
I20260806 03:01:32.125011 11771 raft_consensus.cc:348] T 79b79abb17b843b39436a408ff158fc0 P c7e52d907a814506bfc514dd1d554398 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e52d907a814506bfc514dd1d554398" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 38489 } } peers { permanent_uuid: "1feaea8c263644cebeea7170f5ba31be" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 34281 } } peers { permanent_uuid: "5467a76657c14a18abdfd10227bde360" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 46607 } }
I20260806 03:01:32.125741 11771 raft_consensus.cc:374] T 79b79abb17b843b39436a408ff158fc0 P c7e52d907a814506bfc514dd1d554398 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:01:32.125871 11771 raft_consensus.cc:729] T 79b79abb17b843b39436a408ff158fc0 P c7e52d907a814506bfc514dd1d554398 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7e52d907a814506bfc514dd1d554398, State: Initialized, Role: FOLLOWER
I20260806 03:01:32.126060 11776 tablet_bootstrap.cc:654] T 79b79abb17b843b39436a408ff158fc0 P 1feaea8c263644cebeea7170f5ba31be: Neither blocks nor log segments found. Creating new log.
I20260806 03:01:32.127449 11771 consensus_queue.cc:260] T 79b79abb17b843b39436a408ff158fc0 P c7e52d907a814506bfc514dd1d554398 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e52d907a814506bfc514dd1d554398" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 38489 } } peers { permanent_uuid: "1feaea8c263644cebeea7170f5ba31be" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 34281 } } peers { permanent_uuid: "5467a76657c14a18abdfd10227bde360" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 46607 } }
I20260806 03:01:32.129522 11776 log.cc:824] T 79b79abb17b843b39436a408ff158fc0 P 1feaea8c263644cebeea7170f5ba31be: Log is configured to *not* fsync() on all Append() calls
I20260806 03:01:32.130072 11603 heartbeater.cc:503] Master 127.31.74.254:33855 was elected leader, sending a full tablet report...
I20260806 03:01:32.131415 11771 ts_tablet_manager.cc:1428] T 79b79abb17b843b39436a408ff158fc0 P c7e52d907a814506bfc514dd1d554398: Time spent starting tablet: real 0.021s user 0.017s sys 0.006s
I20260806 03:01:32.135658 11776 tablet_bootstrap.cc:492] T 79b79abb17b843b39436a408ff158fc0 P 1feaea8c263644cebeea7170f5ba31be: No bootstrap required, opened a new log
I20260806 03:01:32.136022 11776 ts_tablet_manager.cc:1397] T 79b79abb17b843b39436a408ff158fc0 P 1feaea8c263644cebeea7170f5ba31be: Time spent bootstrapping tablet: real 0.018s user 0.011s sys 0.004s
I20260806 03:01:32.139132 11775 raft_consensus.cc:348] T 79b79abb17b843b39436a408ff158fc0 P 5467a76657c14a18abdfd10227bde360 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e52d907a814506bfc514dd1d554398" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 38489 } } peers { permanent_uuid: "1feaea8c263644cebeea7170f5ba31be" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 34281 } } peers { permanent_uuid: "5467a76657c14a18abdfd10227bde360" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 46607 } }
I20260806 03:01:32.139741 11775 raft_consensus.cc:374] T 79b79abb17b843b39436a408ff158fc0 P 5467a76657c14a18abdfd10227bde360 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:01:32.139854 11775 raft_consensus.cc:729] T 79b79abb17b843b39436a408ff158fc0 P 5467a76657c14a18abdfd10227bde360 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5467a76657c14a18abdfd10227bde360, State: Initialized, Role: FOLLOWER
I20260806 03:01:32.140515 11775 consensus_queue.cc:260] T 79b79abb17b843b39436a408ff158fc0 P 5467a76657c14a18abdfd10227bde360 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e52d907a814506bfc514dd1d554398" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 38489 } } peers { permanent_uuid: "1feaea8c263644cebeea7170f5ba31be" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 34281 } } peers { permanent_uuid: "5467a76657c14a18abdfd10227bde360" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 46607 } }
I20260806 03:01:32.142743 11775 ts_tablet_manager.cc:1428] T 79b79abb17b843b39436a408ff158fc0 P 5467a76657c14a18abdfd10227bde360: Time spent starting tablet: real 0.018s user 0.015s sys 0.004s
I20260806 03:01:32.151432 11776 raft_consensus.cc:348] T 79b79abb17b843b39436a408ff158fc0 P 1feaea8c263644cebeea7170f5ba31be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e52d907a814506bfc514dd1d554398" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 38489 } } peers { permanent_uuid: "1feaea8c263644cebeea7170f5ba31be" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 34281 } } peers { permanent_uuid: "5467a76657c14a18abdfd10227bde360" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 46607 } }
I20260806 03:01:32.151914 11776 raft_consensus.cc:374] T 79b79abb17b843b39436a408ff158fc0 P 1feaea8c263644cebeea7170f5ba31be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:01:32.152066 11776 raft_consensus.cc:729] T 79b79abb17b843b39436a408ff158fc0 P 1feaea8c263644cebeea7170f5ba31be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1feaea8c263644cebeea7170f5ba31be, State: Initialized, Role: FOLLOWER
I20260806 03:01:32.152742 11776 consensus_queue.cc:260] T 79b79abb17b843b39436a408ff158fc0 P 1feaea8c263644cebeea7170f5ba31be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e52d907a814506bfc514dd1d554398" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 38489 } } peers { permanent_uuid: "1feaea8c263644cebeea7170f5ba31be" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 34281 } } peers { permanent_uuid: "5467a76657c14a18abdfd10227bde360" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 46607 } }
I20260806 03:01:32.154645 11752 heartbeater.cc:503] Master 127.31.74.254:33855 was elected leader, sending a full tablet report...
I20260806 03:01:32.154678 11776 ts_tablet_manager.cc:1428] T 79b79abb17b843b39436a408ff158fc0 P 1feaea8c263644cebeea7170f5ba31be: Time spent starting tablet: real 0.018s user 0.011s sys 0.008s
W20260806 03:01:32.198967 11753 tablet.cc:2369] T 79b79abb17b843b39436a408ff158fc0 P 1feaea8c263644cebeea7170f5ba31be: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 03:01:32.253392 11778 raft_consensus.cc:482] T 79b79abb17b843b39436a408ff158fc0 P c7e52d907a814506bfc514dd1d554398 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:01:32.253805 11778 raft_consensus.cc:504] T 79b79abb17b843b39436a408ff158fc0 P c7e52d907a814506bfc514dd1d554398 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e52d907a814506bfc514dd1d554398" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 38489 } } peers { permanent_uuid: "1feaea8c263644cebeea7170f5ba31be" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 34281 } } peers { permanent_uuid: "5467a76657c14a18abdfd10227bde360" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 46607 } }
I20260806 03:01:32.256307 11778 leader_election.cc:288] T 79b79abb17b843b39436a408ff158fc0 P c7e52d907a814506bfc514dd1d554398 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1feaea8c263644cebeea7170f5ba31be (127.31.74.195:34281), 5467a76657c14a18abdfd10227bde360 (127.31.74.193:46607)
W20260806 03:01:32.259053 11452 tablet.cc:2369] T 79b79abb17b843b39436a408ff158fc0 P 5467a76657c14a18abdfd10227bde360: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 03:01:32.265743 11783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:01:32.256778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:46607 (local address 127.31.74.194:36251)
0806 03:01:32.258281 (+ 1503us) negotiation.cc:107] Waiting for socket to connect
0806 03:01:32.258301 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 03:01:32.258443 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:01:32.259022 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:01:32.259028 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:01:32.259059 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:01:32.259532 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:01:32.259546 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:01:32.261003 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:01:32.261009 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:01:32.263382 (+ 2373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:01:32.263398 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:01:32.264738 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:01:32.264748 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:01:32.264960 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:01:32.265002 (+ 42us) client_negotiation.cc:769] Sending connection context
0806 03:01:32.265117 (+ 115us) client_negotiation.cc:240] Negotiation successful
0806 03:01:32.265268 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1378,"mutex_wait_us":4,"thread_start_us":1099,"threads_started":1}
I20260806 03:01:32.265885 11770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:01:32.257994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.194:36251 (local address 127.31.74.193:46607)
0806 03:01:32.258217 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0806 03:01:32.258223 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 03:01:32.258394 (+ 171us) server_negotiation.cc:407] Connection header received
0806 03:01:32.258615 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:01:32.258623 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:01:32.258700 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:01:32.258838 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:01:32.259928 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:01:32.260842 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:01:32.263568 (+ 2726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:01:32.264375 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:01:32.264525 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:01:32.264582 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0806 03:01:32.265292 (+ 710us) server_negotiation.cc:299] Negotiation successful
0806 03:01:32.265528 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260806 03:01:32.266764 11767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:01:32.256755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.194:51747 (local address 127.31.74.195:34281)
0806 03:01:32.257044 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0806 03:01:32.257054 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 03:01:32.257831 (+ 777us) server_negotiation.cc:407] Connection header received
0806 03:01:32.258214 (+ 383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:01:32.258221 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:01:32.258321 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:01:32.258400 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:01:32.259856 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:01:32.260978 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:01:32.263585 (+ 2607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:01:32.264798 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:01:32.265006 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:01:32.265102 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0806 03:01:32.266317 (+ 1215us) server_negotiation.cc:299] Negotiation successful
0806 03:01:32.266462 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260806 03:01:32.267104 11782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:01:32.256669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.195:34281 (local address 127.31.74.194:51747)
0806 03:01:32.257650 (+ 981us) negotiation.cc:107] Waiting for socket to connect
0806 03:01:32.257694 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 03:01:32.257984 (+ 290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:01:32.258687 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:01:32.258692 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:01:32.258722 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:01:32.259473 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:01:32.259495 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:01:32.261141 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:01:32.261145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:01:32.263392 (+ 2247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:01:32.263405 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:01:32.265488 (+ 2083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:01:32.265498 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:01:32.265755 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:01:32.265798 (+ 43us) client_negotiation.cc:769] Sending connection context
0806 03:01:32.266260 (+ 462us) client_negotiation.cc:240] Negotiation successful
0806 03:01:32.266578 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":765,"mutex_wait_us":42,"thread_start_us":561,"threads_started":1}
I20260806 03:01:32.267477 11403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79b79abb17b843b39436a408ff158fc0" candidate_uuid: "c7e52d907a814506bfc514dd1d554398" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5467a76657c14a18abdfd10227bde360" is_pre_election: true
I20260806 03:01:32.268249 11403 raft_consensus.cc:2454] T 79b79abb17b843b39436a408ff158fc0 P 5467a76657c14a18abdfd10227bde360 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7e52d907a814506bfc514dd1d554398 in term 0.
I20260806 03:01:32.268348 11702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79b79abb17b843b39436a408ff158fc0" candidate_uuid: "c7e52d907a814506bfc514dd1d554398" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1feaea8c263644cebeea7170f5ba31be" is_pre_election: true
I20260806 03:01:32.268998 11702 raft_consensus.cc:2454] T 79b79abb17b843b39436a408ff158fc0 P 1feaea8c263644cebeea7170f5ba31be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7e52d907a814506bfc514dd1d554398 in term 0.
I20260806 03:01:32.269134 11476 leader_election.cc:302] T 79b79abb17b843b39436a408ff158fc0 P c7e52d907a814506bfc514dd1d554398 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5467a76657c14a18abdfd10227bde360, c7e52d907a814506bfc514dd1d554398; no voters:
I20260806 03:01:32.269757 11778 raft_consensus.cc:2781] T 79b79abb17b843b39436a408ff158fc0 P c7e52d907a814506bfc514dd1d554398 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:01:32.269961 11778 raft_consensus.cc:482] T 79b79abb17b843b39436a408ff158fc0 P c7e52d907a814506bfc514dd1d554398 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:01:32.270102 11778 raft_consensus.cc:3037] T 79b79abb17b843b39436a408ff158fc0 P c7e52d907a814506bfc514dd1d554398 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:01:32.273015 11778 raft_consensus.cc:504] T 79b79abb17b843b39436a408ff158fc0 P c7e52d907a814506bfc514dd1d554398 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e52d907a814506bfc514dd1d554398" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 38489 } } peers { permanent_uuid: "1feaea8c263644cebeea7170f5ba31be" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 34281 } } peers { permanent_uuid: "5467a76657c14a18abdfd10227bde360" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 46607 } }
I20260806 03:01:32.273789 11778 leader_election.cc:288] T 79b79abb17b843b39436a408ff158fc0 P c7e52d907a814506bfc514dd1d554398 [CANDIDATE]: Term 1 election: Requested vote from peers 1feaea8c263644cebeea7170f5ba31be (127.31.74.195:34281), 5467a76657c14a18abdfd10227bde360 (127.31.74.193:46607)
I20260806 03:01:32.274461 11403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79b79abb17b843b39436a408ff158fc0" candidate_uuid: "c7e52d907a814506bfc514dd1d554398" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5467a76657c14a18abdfd10227bde360"
I20260806 03:01:32.274469 11702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79b79abb17b843b39436a408ff158fc0" candidate_uuid: "c7e52d907a814506bfc514dd1d554398" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1feaea8c263644cebeea7170f5ba31be"
I20260806 03:01:32.274758 11403 raft_consensus.cc:3037] T 79b79abb17b843b39436a408ff158fc0 P 5467a76657c14a18abdfd10227bde360 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:01:32.274758 11702 raft_consensus.cc:3037] T 79b79abb17b843b39436a408ff158fc0 P 1feaea8c263644cebeea7170f5ba31be [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:01:32.278676 11403 raft_consensus.cc:2454] T 79b79abb17b843b39436a408ff158fc0 P 5467a76657c14a18abdfd10227bde360 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7e52d907a814506bfc514dd1d554398 in term 1.
I20260806 03:01:32.278676 11702 raft_consensus.cc:2454] T 79b79abb17b843b39436a408ff158fc0 P 1feaea8c263644cebeea7170f5ba31be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7e52d907a814506bfc514dd1d554398 in term 1.
I20260806 03:01:32.279333 11478 leader_election.cc:302] T 79b79abb17b843b39436a408ff158fc0 P c7e52d907a814506bfc514dd1d554398 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1feaea8c263644cebeea7170f5ba31be, c7e52d907a814506bfc514dd1d554398; no voters:
I20260806 03:01:32.279898 11778 raft_consensus.cc:2781] T 79b79abb17b843b39436a408ff158fc0 P c7e52d907a814506bfc514dd1d554398 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:01:32.280601 11778 raft_consensus.cc:686] T 79b79abb17b843b39436a408ff158fc0 P c7e52d907a814506bfc514dd1d554398 [term 1 LEADER]: Becoming Leader. State: Replica: c7e52d907a814506bfc514dd1d554398, State: Running, Role: LEADER
I20260806 03:01:32.281466 11778 consensus_queue.cc:237] T 79b79abb17b843b39436a408ff158fc0 P c7e52d907a814506bfc514dd1d554398 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e52d907a814506bfc514dd1d554398" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 38489 } } peers { permanent_uuid: "1feaea8c263644cebeea7170f5ba31be" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 34281 } } peers { permanent_uuid: "5467a76657c14a18abdfd10227bde360" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 46607 } }
I20260806 03:01:32.288846 11228 catalog_manager.cc:5495] T 79b79abb17b843b39436a408ff158fc0 P c7e52d907a814506bfc514dd1d554398 reported cstate change: term changed from 0 to 1, leader changed from <none> to c7e52d907a814506bfc514dd1d554398 (127.31.74.194). New cstate: current_term: 1 leader_uuid: "c7e52d907a814506bfc514dd1d554398" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e52d907a814506bfc514dd1d554398" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 38489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1feaea8c263644cebeea7170f5ba31be" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 34281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5467a76657c14a18abdfd10227bde360" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 46607 } health_report { overall_health: UNKNOWN } } }
I20260806 03:01:32.327777 11766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:01:32.318518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34908 (local address 127.31.74.194:38489)
0806 03:01:32.318865 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0806 03:01:32.318873 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 03:01:32.319180 (+ 307us) server_negotiation.cc:407] Connection header received
0806 03:01:32.319927 (+ 747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:01:32.319934 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:01:32.320113 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:01:32.320484 (+ 371us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 03:01:32.321168 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:01:32.322063 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:01:32.324372 (+ 2309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:01:32.324930 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:01:32.325072 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:01:32.326184 (+ 1112us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 03:01:32.326879 (+ 695us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 03:01:32.327017 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0806 03:01:32.327304 (+ 287us) server_negotiation.cc:299] Negotiation successful
0806 03:01:32.327397 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146}
I20260806 03:01:32.339093 11403 raft_consensus.cc:1261] T 79b79abb17b843b39436a408ff158fc0 P 5467a76657c14a18abdfd10227bde360 [term 1 FOLLOWER]: Refusing update from remote peer c7e52d907a814506bfc514dd1d554398: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:01:32.339591 11702 raft_consensus.cc:1261] T 79b79abb17b843b39436a408ff158fc0 P 1feaea8c263644cebeea7170f5ba31be [term 1 FOLLOWER]: Refusing update from remote peer c7e52d907a814506bfc514dd1d554398: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:01:32.340911 11784 consensus_queue.cc:1035] T 79b79abb17b843b39436a408ff158fc0 P c7e52d907a814506bfc514dd1d554398 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5467a76657c14a18abdfd10227bde360" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 46607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:01:32.341552 11778 consensus_queue.cc:1035] T 79b79abb17b843b39436a408ff158fc0 P c7e52d907a814506bfc514dd1d554398 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1feaea8c263644cebeea7170f5ba31be" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 34281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 03:01:32.360318 11604 tablet.cc:2369] T 79b79abb17b843b39436a408ff158fc0 P c7e52d907a814506bfc514dd1d554398: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 03:01:32.366122 11788 mvcc.cc:205] Tried to move back new op lower bound from 7315395757407236096 to 7315395757191532544. Current Snapshot: MvccSnapshot[applied={T|T < 7315395757407236096}]
I20260806 03:01:32.392566 11229 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 03:01:32.393038 11229 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 03:01:32.397975 11229 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8d55d44dee6c411f9dbc7d74f91bb168: Sending DeleteTablet for 3 replicas of tablet 79b79abb17b843b39436a408ff158fc0
I20260806 03:01:32.399582 11383 tablet_service.cc:1514] Processing DeleteTablet for tablet 79b79abb17b843b39436a408ff158fc0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:01:32 UTC) from {username='kudu', principal='kudu/127.31.74.254@KRBTEST.COM'} at 127.0.0.1:36358
I20260806 03:01:32.399516 11531 tablet_service.cc:1514] Processing DeleteTablet for tablet 79b79abb17b843b39436a408ff158fc0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:01:32 UTC) from {username='kudu', principal='kudu/127.31.74.254@KRBTEST.COM'} at 127.0.0.1:34906
I20260806 03:01:32.399794 11678 tablet_service.cc:1514] Processing DeleteTablet for tablet 79b79abb17b843b39436a408ff158fc0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:01:32 UTC) from {username='kudu', principal='kudu/127.31.74.254@KRBTEST.COM'} at 127.0.0.1:41816
I20260806 03:01:32.407936 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 11310
I20260806 03:01:32.413844 11798 tablet_replica.cc:330] T 79b79abb17b843b39436a408ff158fc0 P 1feaea8c263644cebeea7170f5ba31be: stopping tablet replica
I20260806 03:01:32.415179 11798 raft_consensus.cc:2229] T 79b79abb17b843b39436a408ff158fc0 P 1feaea8c263644cebeea7170f5ba31be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:01:32.416404 11798 raft_consensus.cc:2258] T 79b79abb17b843b39436a408ff158fc0 P 1feaea8c263644cebeea7170f5ba31be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:01:32.417773 11796 tablet_replica.cc:330] T 79b79abb17b843b39436a408ff158fc0 P c7e52d907a814506bfc514dd1d554398: stopping tablet replica
I20260806 03:01:32.420234 11797 tablet_replica.cc:330] T 79b79abb17b843b39436a408ff158fc0 P 5467a76657c14a18abdfd10227bde360: stopping tablet replica
I20260806 03:01:32.420490 11796 raft_consensus.cc:2229] T 79b79abb17b843b39436a408ff158fc0 P c7e52d907a814506bfc514dd1d554398 [term 1 LEADER]: Raft consensus shutting down.
I20260806 03:01:32.421172 11797 raft_consensus.cc:2229] T 79b79abb17b843b39436a408ff158fc0 P 5467a76657c14a18abdfd10227bde360 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:01:32.421451 11798 ts_tablet_manager.cc:1905] T 79b79abb17b843b39436a408ff158fc0 P 1feaea8c263644cebeea7170f5ba31be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:01:32.421870 11796 raft_consensus.cc:2258] T 79b79abb17b843b39436a408ff158fc0 P c7e52d907a814506bfc514dd1d554398 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:01:32.421955 11797 raft_consensus.cc:2258] T 79b79abb17b843b39436a408ff158fc0 P 5467a76657c14a18abdfd10227bde360 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:01:32.426133 11797 ts_tablet_manager.cc:1905] T 79b79abb17b843b39436a408ff158fc0 P 5467a76657c14a18abdfd10227bde360: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:01:32.435434 11796 ts_tablet_manager.cc:1905] T 79b79abb17b843b39436a408ff158fc0 P c7e52d907a814506bfc514dd1d554398: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 06 03:01:32 dist-test-slave-dhph krb5kdc[10726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.74.193@KRBTEST.COM
I20260806 03:01:32.439509 11797 ts_tablet_manager.cc:1918] T 79b79abb17b843b39436a408ff158fc0 P 5467a76657c14a18abdfd10227bde360: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 03:01:32.439822 11797 log.cc:1196] T 79b79abb17b843b39436a408ff158fc0 P 5467a76657c14a18abdfd10227bde360: Deleting WAL directory at /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/ts-0/wal/wals/79b79abb17b843b39436a408ff158fc0
I20260806 03:01:32.441263 11797 ts_tablet_manager.cc:1939] T 79b79abb17b843b39436a408ff158fc0 P 5467a76657c14a18abdfd10227bde360: Deleting consensus metadata
I20260806 03:01:32.444398 11212 catalog_manager.cc:4841] TS 5467a76657c14a18abdfd10227bde360 (127.31.74.193:46607): tablet 79b79abb17b843b39436a408ff158fc0 (table test-table [id=5904dc936f3c4334bdd2cc760e081627]) successfully deleted
I20260806 03:01:32.449115 11798 ts_tablet_manager.cc:1918] T 79b79abb17b843b39436a408ff158fc0 P 1feaea8c263644cebeea7170f5ba31be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 03:01:32.449481 11798 log.cc:1196] T 79b79abb17b843b39436a408ff158fc0 P 1feaea8c263644cebeea7170f5ba31be: Deleting WAL directory at /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/ts-2/wal/wals/79b79abb17b843b39436a408ff158fc0
I20260806 03:01:32.450320 11798 ts_tablet_manager.cc:1939] T 79b79abb17b843b39436a408ff158fc0 P 1feaea8c263644cebeea7170f5ba31be: Deleting consensus metadata
I20260806 03:01:32.451159 11796 ts_tablet_manager.cc:1918] T 79b79abb17b843b39436a408ff158fc0 P c7e52d907a814506bfc514dd1d554398: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 03:01:32.451391 11796 log.cc:1196] T 79b79abb17b843b39436a408ff158fc0 P c7e52d907a814506bfc514dd1d554398: Deleting WAL directory at /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785985105255475-32043-0/minicluster-data/ts-1/wal/wals/79b79abb17b843b39436a408ff158fc0
I20260806 03:01:32.452071 11796 ts_tablet_manager.cc:1939] T 79b79abb17b843b39436a408ff158fc0 P c7e52d907a814506bfc514dd1d554398: Deleting consensus metadata
I20260806 03:01:32.453006 11770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:01:32.410020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36364 (local address 127.31.74.193:46607)
0806 03:01:32.410388 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0806 03:01:32.410398 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 03:01:32.410782 (+ 384us) server_negotiation.cc:407] Connection header received
0806 03:01:32.412083 (+ 1301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:01:32.412092 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:01:32.412248 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:01:32.412360 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:01:32.422571 (+ 10211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:01:32.423850 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:01:32.428878 (+ 5028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:01:32.429692 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:01:32.429843 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:01:32.439842 (+ 9999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:01:32.439862 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:01:32.439888 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:01:32.439985 (+ 97us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:01:32.446864 (+ 6879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:01:32.448586 (+ 1722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:01:32.448600 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:01:32.448604 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:01:32.448837 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:01:32.449446 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:01:32.449452 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:01:32.449456 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:01:32.450021 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:01:32.450383 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0806 03:01:32.452058 (+ 1675us) server_negotiation.cc:299] Negotiation successful
0806 03:01:32.452514 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260806 03:01:32.453003 11214 catalog_manager.cc:4841] TS 1feaea8c263644cebeea7170f5ba31be (127.31.74.195:34281): tablet 79b79abb17b843b39436a408ff158fc0 (table test-table [id=5904dc936f3c4334bdd2cc760e081627]) successfully deleted
I20260806 03:01:32.454137 11445 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 03:01:32.454794 11213 catalog_manager.cc:4841] TS c7e52d907a814506bfc514dd1d554398 (127.31.74.194:38489): tablet 79b79abb17b843b39436a408ff158fc0 (table test-table [id=5904dc936f3c4334bdd2cc760e081627]) successfully deleted
I20260806 03:01:32.626775 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 11310
I20260806 03:01:32.658641 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 11457
Aug 06 03:01:32 dist-test-slave-dhph krb5kdc[10726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.74.194@KRBTEST.COM
I20260806 03:01:32.690905 11766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:01:32.659939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34918 (local address 127.31.74.194:38489)
0806 03:01:32.660181 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0806 03:01:32.660188 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 03:01:32.660217 (+ 29us) server_negotiation.cc:407] Connection header received
0806 03:01:32.660452 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:01:32.660459 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:01:32.660565 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:01:32.660714 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:01:32.662517 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:01:32.664538 (+ 2021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:01:32.667890 (+ 3352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:01:32.669440 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:01:32.670212 (+ 772us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:01:32.677637 (+ 7425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:01:32.677663 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:01:32.677687 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:01:32.677761 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:01:32.686104 (+ 8343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:01:32.687222 (+ 1118us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:01:32.687234 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:01:32.687248 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:01:32.687478 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:01:32.688107 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:01:32.688114 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:01:32.688118 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:01:32.688717 (+ 599us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:01:32.689107 (+ 390us) server_negotiation.cc:1035] Waiting for connection context
0806 03:01:32.690090 (+ 983us) server_negotiation.cc:299] Negotiation successful
0806 03:01:32.690402 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260806 03:01:32.691884 11598 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 03:01:32.838470 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 11457
I20260806 03:01:32.865173 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 11609
Aug 06 03:01:32 dist-test-slave-dhph krb5kdc[10726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.74.195@KRBTEST.COM
I20260806 03:01:32.884855 11802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:01:32.866333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41818 (local address 127.31.74.195:34281)
0806 03:01:32.866980 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0806 03:01:32.866985 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 03:01:32.867006 (+ 21us) server_negotiation.cc:407] Connection header received
0806 03:01:32.867101 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:01:32.867107 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:01:32.867231 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:01:32.867387 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:01:32.868875 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:01:32.869861 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:01:32.871486 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:01:32.871951 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:01:32.872067 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:01:32.876889 (+ 4822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:01:32.876909 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:01:32.876926 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:01:32.876979 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:01:32.880362 (+ 3383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:01:32.881259 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:01:32.881268 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:01:32.881271 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:01:32.881445 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:01:32.882340 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:01:32.882349 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:01:32.882352 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:01:32.883358 (+ 1006us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:01:32.883643 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0806 03:01:32.884119 (+ 476us) server_negotiation.cc:299] Negotiation successful
0806 03:01:32.884363 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":350,"threads_started":1}
I20260806 03:01:32.885751 11747 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 03:01:32.996512 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 11609
I20260806 03:01:33.019738 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 11195
I20260806 03:01:33.021919 11259 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 03:01:33.159068 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 11195
I20260806 03:01:33.196985 32043 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260806 03:01:33.545415 32043 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260806 03:01:33.546036 32043 mini_ranger.cc:66] Stopping Ranger...
I20260806 03:01:33.904656 32043 mini_ranger.cc:68] Stopped Ranger
2026-08-06 03:01:33.905 UTC [10756] LOG: received smart shutdown request
2026-08-06 03:01:33.909 UTC [10756] LOG: background worker "logical replication launcher" (PID 10765) exited with exit code 1
2026-08-06 03:01:33.910 UTC [10760] LOG: shutting down
2026-08-06 03:01:34.211 UTC [10756] LOG: database system is shut down
2026-08-06T03:01:34Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (70747 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 03:01:34 dist-test-slave-dhph krb5kdc[11814](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 03:01:34 dist-test-slave-dhph krb5kdc[11814](info): set up 2 sockets
Aug 06 03:01:34 dist-test-slave-dhph krb5kdc[11814](info): commencing operation
krb5kdc: starting...
W20260806 03:01:35.752853 32043 mini_kdc.cc:121] Time spent starting KDC: real 1.300s user 0.000s sys 0.052s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/krb5kdc/test-user.keytab.
Aug 06 03:01:36 dist-test-slave-dhph krb5kdc[11814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T03:01:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:01:36Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.31.74.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.31.74.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.31.74.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/krb5kdc/rangeradmin_127.31.74.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.31.74.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/krb5kdc/rangeradmin_127.31.74.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.31.74.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.31.74.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.31.74.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/krb5kdc/rangerlookup_127.31.74.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.31.74.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/krb5kdc/rangerlookup_127.31.74.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.31.74.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.31.74.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.31.74.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/krb5kdc/HTTP_127.31.74.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.31.74.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/krb5kdc/HTTP_127.31.74.212@KRBTEST.COM.keytab.
I20260806 03:01:36.327778 32043 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskGJ8w1x/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/postgres -l logfile start
2026-08-06 03:01:40.020 UTC [11848] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-06 03:01:40.020 UTC [11848] LOG: listening on IPv4 address "127.31.74.212", port 39013
2026-08-06 03:01:40.027 UTC [11848] LOG: listening on Unix socket "/tmp/.s.PGSQL.39013"
2026-08-06 03:01:40.059 UTC [11851] LOG: database system was shut down at 2026-08-06 03:01:37 UTC
2026-08-06 03:01:40.064 UTC [11848] LOG: database system is ready to accept connections
I20260806 03:01:41.223907 32043 mini_postgres.cc:96] Postgres bound to 39013
2026-08-06 03:01:41.261 UTC [11859] FATAL: database "slave" does not exist
127.31.74.212:39013 - accepting connections
I20260806 03:01:41.262456 32043 mini_ranger.cc:161] Starting Ranger...
I20260806 03:01:41.304512 32043 mini_ranger.cc:84] Created miniranger Postgres user
I20260806 03:01:42.192392 32043 mini_ranger.cc:87] Created ranger Postgres database
I20260806 03:01:42.192556 32043 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ranger-admin
2026-08-06 03:01:42,560 [I] DB FLAVOR :POSTGRES
2026-08-06 03:01:42,561 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 03:01:42,561 [I] Checking connection..
2026-08-06 03:01:42,561 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJ8w1x/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGJ8w1x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.74.212:39013/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-06 03:01:42,886 [I] Checking connection passed.
2026-08-06 03:01:42,886 [I] --------- Verifying version history table ---------
2026-08-06 03:01:42,887 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJ8w1x/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGJ8w1x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.74.212:39013/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 03:01:43,238 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 03:01:43,239 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJ8w1x/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGJ8w1x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.74.212:39013/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 03:01:43,637 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 03:01:43,638 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-06 03:01:43,639 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJ8w1x/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGJ8w1x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.74.212:39013/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskGJ8w1x/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-06 03:01:43.983 UTC [11929] WARNING: there is no transaction in progress
2026-08-06 03:01:43,995 [I] create_dbversion_catalog.sql file imported successfully
2026-08-06 03:01:43,996 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJ8w1x/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGJ8w1x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.74.212:39013/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 03:01:44,338 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-06 03:01:44,338 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 03:01:44,339 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJ8w1x/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGJ8w1x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.74.212:39013/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-06 03:01:44,679 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJ8w1x/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGJ8w1x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.74.212:39013/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-06 03:01:44,996 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJ8w1x/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGJ8w1x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.74.212:39013/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-dhph.c.gcp-upstream.internal','N') ;"
2026-08-06 03:01:45,330 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJ8w1x/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGJ8w1x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.74.212:39013/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 03:01:45,664 [I] Table x_portal_user does not exist in database ranger
2026-08-06 03:01:45,664 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJ8w1x/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGJ8w1x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.74.212:39013/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 03:01:45,982 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-06 03:01:45,983 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJ8w1x/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGJ8w1x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.74.212:39013/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 03:01:46,304 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJ8w1x/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGJ8w1x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.74.212:39013/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 03:01:46,634 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-06 03:01:46,634 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJ8w1x/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGJ8w1x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.74.212:39013/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskGJ8w1x/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-06 03:01:48.004 UTC [12043] WARNING: there is no transaction in progress
2026-08-06 03:01:48.033 UTC [12043] WARNING: there is no transaction in progress
2026-08-06 03:01:48.059 UTC [12043] WARNING: there is no transaction in progress
2026-08-06 03:01:48.079 UTC [12043] WARNING: there is no transaction in progress
2026-08-06 03:01:48.104 UTC [12043] WARNING: there is no transaction in progress
2026-08-06 03:01:48.134 UTC [12043] WARNING: there is no transaction in progress
2026-08-06 03:01:48.222 UTC [12043] WARNING: there is no transaction in progress
2026-08-06 03:01:48.233 UTC [12043] WARNING: there is no transaction in progress
2026-08-06 03:01:48.255 UTC [12043] WARNING: there is no transaction in progress
2026-08-06 03:01:48.270 UTC [12043] WARNING: there is no transaction in progress
2026-08-06 03:01:48.284 UTC [12043] WARNING: there is no transaction in progress
2026-08-06 03:01:48.303 UTC [12043] WARNING: there is no transaction in progress
2026-08-06 03:01:48.316 UTC [12043] WARNING: there is no transaction in progress
2026-08-06 03:01:48.325 UTC [12043] WARNING: there is no transaction in progress
2026-08-06 03:01:48.336 UTC [12043] WARNING: there is no transaction in progress
2026-08-06 03:01:48.347 UTC [12043] WARNING: there is no transaction in progress
2026-08-06 03:01:49,506 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-06 03:01:49,506 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJ8w1x/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGJ8w1x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.74.212:39013/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-06 03:01:49,843 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-06 03:01:49,844 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJ8w1x/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGJ8w1x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.74.212:39013/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 03:01:50,183 [I] Table x_portal_user already exists in database 'ranger'
2026-08-06 03:01:50,183 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJ8w1x/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGJ8w1x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.74.212:39013/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 03:01:50,474 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-06 03:01:50,474 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJ8w1x/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGJ8w1x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.74.212:39013/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 03:01:50,768 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJ8w1x/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGJ8w1x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.74.212:39013/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 03:01:51,080 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJ8w1x/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGJ8w1x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.74.212:39013/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-dhph.c.gcp-upstream.internal';"
2026-08-06 03:01:51,371 [I] CORE_DB_SCHEMA import status has been updated
2026-08-06 03:01:51,372 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJ8w1x/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGJ8w1x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.74.212:39013/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-06 03:01:51,661 [I] DB_PATCHES have already been applied
I20260806 03:01:51.668829 32043 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ranger-admin:/tmp/dist-test-taskGJ8w1x/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskGJ8w1x/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskGJ8w1x/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGJ8w1x/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260806 03:01:51.668936 32043 mini_ranger.cc:193] Using host: 127.31.74.212
I20260806 03:01:51.694998 32043 mini_ranger.cc:227] Ranger admin URL: http://127.31.74.212:44741
Aug 06, 2026 3:01:52 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 3:01:52 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 3:01:52 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ranger-admin/ews/webapp
Aug 06, 2026 3:01:52 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 06, 2026 3:01:52 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 06, 2026 3:01:52 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ranger-admin/ews/webapp].
Aug 06, 2026 3:01:52 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260806 03:01:52.987929 32043 mini_ranger.cc:160] Time spent starting Ranger: real 11.726s user 0.002s sys 0.078s
Aug 06, 2026 3:02:20 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 06, 2026 3:02:20 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Aug 06, 2026 3:02:20 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 06, 2026 3:02:20 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 06, 2026 3:02:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 06, 2026 3:02:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 06, 2026 3:02:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 06, 2026 3:02:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 06, 2026 3:02:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 06, 2026 3:02:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 06, 2026 3:02:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 06, 2026 3:02:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 06, 2026 3:02:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 06, 2026 3:02:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 06, 2026 3:02:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 06, 2026 3:02:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 06, 2026 3:02:20 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 06, 2026 3:02:22 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260806 03:02:23.080225 32043 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.31.74.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.31.74.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.31.74.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/krb5kdc/rangerkms_127.31.74.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.31.74.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/krb5kdc/rangerkms_127.31.74.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.31.74.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.31.74.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.31.74.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/krb5kdc/HTTP_127.31.74.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.31.74.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/krb5kdc/HTTP_127.31.74.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 06 03:02:23 dist-test-slave-dhph krb5kdc[11814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985343, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260806 03:02:23.383102 32043 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260806 03:02:23.423753 32043 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260806 03:02:24.901480 32043 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260806 03:02:24.901633 32043 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ranger-kms
I20260806 03:02:24.901684 32043 mini_ranger_kms.cc:227] Using postgres at 127.31.74.212:39013
2026-08-06 03:02:25,485 [I] DB FLAVOR :POSTGRES
2026-08-06 03:02:25,486 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 03:02:25,486 [I] Checking connection
2026-08-06 03:02:25,486 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJ8w1x/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.74.212:39013/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-06 03:02:25,837 [I] connection success
2026-08-06 03:02:25,837 [I] --------- Verifying Ranger DB tables ---------
2026-08-06 03:02:25,837 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-06 03:02:25,838 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJ8w1x/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.74.212:39013/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-06 03:02:26,186 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-06 03:02:26,186 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 03:02:26,187 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-06 03:02:26,187 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGJ8w1x/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.74.212:39013/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-06 03:02:26,574 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260806 03:02:27.387130 32043 mini_ranger_kms.cc:310] Created kms service
I20260806 03:02:27.689879 32043 mini_ranger_kms.cc:326] Created kudu user
I20260806 03:02:27.763111 32043 mini_ranger_kms.cc:343] Created rangerkms user
Aug 06 03:02:27 dist-test-slave-dhph krb5kdc[11814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985343, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.31.74.212@KRBTEST.COM
I20260806 03:02:28.370872 32043 mini_ranger_kms.cc:384] Added ranger policy
I20260806 03:02:28.371759 32043 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ranger-kms:/tmp/dist-test-taskGJ8w1x/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGJ8w1x/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskGJ8w1x/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260806 03:02:28.371850 32043 mini_ranger_kms.cc:240] Using host: 127.31.74.212
I20260806 03:02:28.413228 32043 mini_ranger_kms.cc:276] Ranger KMS PID: 12236
I20260806 03:02:28.413393 32043 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.31.74.212:59733
Aug 06, 2026 3:02:29 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 3:02:30 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 3:02:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 06, 2026 3:02:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 06, 2026 3:02:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ranger-kms/ews/webapp].
Aug 06, 2026 3:02:30 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260806 03:02:31.229211 32043 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 7.846s user 0.125s sys 0.509s
I20260806 03:02:31.229487 32043 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260806 03:02:31.229710 32043 mini_ranger_kms.cc:401] 127.31.74.212:59733/kms/v1/keys
Aug 06 03:02:31 dist-test-slave-dhph krb5kdc[11814](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1785985351, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.74.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 06 03:02:31 dist-test-slave-dhph krb5kdc[11814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985351, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.74.212@KRBTEST.COM for HTTP/127.31.74.212@KRBTEST.COM
Aug 06 03:02:32 dist-test-slave-dhph krb5kdc[11814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985351, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.74.212@KRBTEST.COM for HTTP/127.31.74.212@KRBTEST.COM
Aug 06 03:02:35 dist-test-slave-dhph krb5kdc[11814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.31.74.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab.
Entry for principal kudu/127.31.74.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab.
I20260806 03:02:35.444475 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:36839
--webserver_interface=127.31.74.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab
--principal=kudu/127.31.74.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:35599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.74.254:36839
--ranger_config_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.74.212:59733/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:02:35.846330 12289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:02:35.846812 12289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:02:35.846987 12289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:02:35.855293 12289 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:02:35.855494 12289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:02:35.855615 12289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:02:35.855724 12289 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:02:35.855824 12289 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:02:35.868741 12289 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:35599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/master-0/wal
--ranger_kms_url=127.31.74.212:59733/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.31.74.254:36839
--ranger_config_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab
--principal=kudu/127.31.74.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:36839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.74.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:02:35.870438 12289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:02:35.872573 12289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:02:35.887537 12295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:02:35.887692 12297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:02:35.887537 12294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:02:35.888003 12289 server_base.cc:1034] running on GCE node
I20260806 03:02:35.889196 12289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:02:35.891234 12289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:02:35.892655 12289 hybrid_clock.cc:648] HybridClock initialized: now 1785985355892541 us; error 91 us; skew 500 ppm
I20260806 03:02:35.893188 12289 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 03:02:35 dist-test-slave-dhph krb5kdc[11814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985355, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:02:35.897815 12289 init.cc:377] Logged in from keytab as kudu/127.31.74.254@KRBTEST.COM (short username kudu)
I20260806 03:02:35.900308 12289 webserver.cc:460] Webserver started at http://127.31.74.254:36475/ using document root <none> and password file <none>
I20260806 03:02:35.901101 12289 fs_manager.cc:359] Metadata directory not provided
I20260806 03:02:35.901504 12289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:02:35.901906 12289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 03:02:35 dist-test-slave-dhph krb5kdc[11814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985355, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.254@KRBTEST.COM for HTTP/127.31.74.212@KRBTEST.COM
I20260806 03:02:35.972582 12289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/master-0/data/instance:
uuid: "60efcc97d32b4aaa968cbb910c59282e"
format_stamp: "Formatted at 2026-08-06 03:02:35 on dist-test-slave-dhph"
server_key: "09e260575ed0a29813ff24f49c0fa370"
server_key_iv: "77a7af5755ec66ece218a251ed9eb13b"
server_key_version: "kuduclusterkey@0"
I20260806 03:02:35.973595 12289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/master-0/wal/instance:
uuid: "60efcc97d32b4aaa968cbb910c59282e"
format_stamp: "Formatted at 2026-08-06 03:02:35 on dist-test-slave-dhph"
server_key: "09e260575ed0a29813ff24f49c0fa370"
server_key_iv: "77a7af5755ec66ece218a251ed9eb13b"
server_key_version: "kuduclusterkey@0"
I20260806 03:02:35.980156 12289 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260806 03:02:35.999751 12304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:02:36.001477 12289 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.002s
I20260806 03:02:36.001711 12289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/master-0/wal
uuid: "60efcc97d32b4aaa968cbb910c59282e"
format_stamp: "Formatted at 2026-08-06 03:02:35 on dist-test-slave-dhph"
server_key: "09e260575ed0a29813ff24f49c0fa370"
server_key_iv: "77a7af5755ec66ece218a251ed9eb13b"
server_key_version: "kuduclusterkey@0"
I20260806 03:02:36.001986 12289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:02:36.023492 12289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:02:36.030929 12289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:02:36.031340 12289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:02:36.056337 12289 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.254:36839
I20260806 03:02:36.056337 12358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.254:36839 every 8 connection(s)
I20260806 03:02:36.058730 12289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
I20260806 03:02:36.062438 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 12289
I20260806 03:02:36.062716 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/master-0/wal/instance
I20260806 03:02:36.063601 32043 external_mini_cluster.cc:1428] Setting key 23c84a7d74fa88b239d50edeb625895a
I20260806 03:02:36.063879 12359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:02:36.084816 12359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60efcc97d32b4aaa968cbb910c59282e: Bootstrap starting.
I20260806 03:02:36.089179 12359 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60efcc97d32b4aaa968cbb910c59282e: Neither blocks nor log segments found. Creating new log.
I20260806 03:02:36.090497 12359 log.cc:824] T 00000000000000000000000000000000 P 60efcc97d32b4aaa968cbb910c59282e: Log is configured to *not* fsync() on all Append() calls
I20260806 03:02:36.094233 12359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60efcc97d32b4aaa968cbb910c59282e: No bootstrap required, opened a new log
Aug 06 03:02:36 dist-test-slave-dhph krb5kdc[11814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 03:02:36.107761 12359 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60efcc97d32b4aaa968cbb910c59282e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60efcc97d32b4aaa968cbb910c59282e" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36839 } }
I20260806 03:02:36.108290 12359 raft_consensus.cc:374] T 00000000000000000000000000000000 P 60efcc97d32b4aaa968cbb910c59282e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:02:36.108532 12359 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60efcc97d32b4aaa968cbb910c59282e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60efcc97d32b4aaa968cbb910c59282e, State: Initialized, Role: FOLLOWER
I20260806 03:02:36.109314 12359 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60efcc97d32b4aaa968cbb910c59282e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60efcc97d32b4aaa968cbb910c59282e" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36839 } }
I20260806 03:02:36.109642 12359 raft_consensus.cc:388] T 00000000000000000000000000000000 P 60efcc97d32b4aaa968cbb910c59282e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:02:36.109823 12359 raft_consensus.cc:482] T 00000000000000000000000000000000 P 60efcc97d32b4aaa968cbb910c59282e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:02:36.110013 12359 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60efcc97d32b4aaa968cbb910c59282e [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:02:36.111630 12362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:36.065875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53246 (local address 127.31.74.254:36839)
0806 03:02:36.082267 (+ 16392us) server_negotiation.cc:206] Beginning negotiation
0806 03:02:36.082287 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 03:02:36.082344 (+ 57us) server_negotiation.cc:407] Connection header received
0806 03:02:36.085067 (+ 2723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:02:36.085109 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:02:36.085670 (+ 561us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:02:36.086129 (+ 459us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:02:36.088247 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:36.090546 (+ 2299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:36.092619 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:36.093663 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:36.093861 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:02:36.100322 (+ 6461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:02:36.100337 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:02:36.100361 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:02:36.100421 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:02:36.105382 (+ 4961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:02:36.106567 (+ 1185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:02:36.106602 (+ 35us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:02:36.106613 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:02:36.106856 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:02:36.107362 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:02:36.107367 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:02:36.107370 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:02:36.108170 (+ 800us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:02:36.108478 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0806 03:02:36.109204 (+ 726us) server_negotiation.cc:299] Negotiation successful
0806 03:02:36.109787 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16099,"thread_start_us":15826,"threads_started":1}
I20260806 03:02:36.114037 12359 raft_consensus.cc:504] T 00000000000000000000000000000000 P 60efcc97d32b4aaa968cbb910c59282e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60efcc97d32b4aaa968cbb910c59282e" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36839 } }
I20260806 03:02:36.114739 12359 leader_election.cc:302] T 00000000000000000000000000000000 P 60efcc97d32b4aaa968cbb910c59282e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 60efcc97d32b4aaa968cbb910c59282e; no voters:
I20260806 03:02:36.116297 12359 leader_election.cc:288] T 00000000000000000000000000000000 P 60efcc97d32b4aaa968cbb910c59282e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:02:36.116353 12364 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 60efcc97d32b4aaa968cbb910c59282e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:02:36.117563 12364 raft_consensus.cc:686] T 00000000000000000000000000000000 P 60efcc97d32b4aaa968cbb910c59282e [term 1 LEADER]: Becoming Leader. State: Replica: 60efcc97d32b4aaa968cbb910c59282e, State: Running, Role: LEADER
I20260806 03:02:36.118255 12364 consensus_queue.cc:237] T 00000000000000000000000000000000 P 60efcc97d32b4aaa968cbb910c59282e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60efcc97d32b4aaa968cbb910c59282e" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36839 } }
I20260806 03:02:36.120008 12359 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60efcc97d32b4aaa968cbb910c59282e [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:02:36.121632 12365 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60efcc97d32b4aaa968cbb910c59282e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 60efcc97d32b4aaa968cbb910c59282e. Latest consensus state: current_term: 1 leader_uuid: "60efcc97d32b4aaa968cbb910c59282e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60efcc97d32b4aaa968cbb910c59282e" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36839 } } }
I20260806 03:02:36.122078 12365 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60efcc97d32b4aaa968cbb910c59282e [sys.catalog]: This master's current role is: LEADER
I20260806 03:02:36.121606 12366 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60efcc97d32b4aaa968cbb910c59282e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "60efcc97d32b4aaa968cbb910c59282e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60efcc97d32b4aaa968cbb910c59282e" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36839 } } }
I20260806 03:02:36.122622 12366 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60efcc97d32b4aaa968cbb910c59282e [sys.catalog]: This master's current role is: LEADER
I20260806 03:02:36.124445 12369 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:02:36.131173 12369 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:02:36.141284 12369 catalog_manager.cc:1347] Generated new cluster ID: a8bc0021265e47e7915d531336956ccc
I20260806 03:02:36.141428 12369 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:02:36.143668 12359 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260806 03:02:36.191476 12369 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:02:36.193964 12369 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:02:36.214730 12369 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 60efcc97d32b4aaa968cbb910c59282e: Generated new TSK 0
I20260806 03:02:36.216137 12369 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.74.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.193.keytab.
Entry for principal kudu/127.31.74.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.193.keytab.
I20260806 03:02:37.099891 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.74.212:59733/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.193:0
--local_ip_for_outbound_sockets=127.31.74.193
--webserver_interface=127.31.74.193
--webserver_port=0
--tserver_master_addrs=127.31.74.254:36839
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.193.keytab
--principal=kudu/127.31.74.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:35599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 03:02:37 dist-test-slave-dhph krb5kdc[11814](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1785985357, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260806 03:02:37.577301 12407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:02:37.577766 12407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:02:37.578282 12407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:02:37.594079 12407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:02:37.594491 12407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:02:37.594792 12407 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.193
I20260806 03:02:37.616099 12407 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:35599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.31.74.212:59733/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.193.keytab
--principal=kudu/127.31.74.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.74.193
--webserver_port=0
--tserver_master_addrs=127.31.74.254:36839
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.193
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:02:37.617898 12407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:02:37.619728 12407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:02:37.671046 12418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:02:37.672060 12419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:02:37.680949 12407 server_base.cc:1034] running on GCE node
W20260806 03:02:37.681365 12421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:02:37.682343 12407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:02:37.684230 12407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:02:37.687716 12407 hybrid_clock.cc:648] HybridClock initialized: now 1785985357687624 us; error 63 us; skew 500 ppm
I20260806 03:02:37.688282 12407 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 03:02:37 dist-test-slave-dhph krb5kdc[11814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985357, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:02:37.693403 12407 init.cc:377] Logged in from keytab as kudu/127.31.74.193@KRBTEST.COM (short username kudu)
I20260806 03:02:37.696165 12407 webserver.cc:460] Webserver started at http://127.31.74.193:42683/ using document root <none> and password file <none>
I20260806 03:02:37.697353 12407 fs_manager.cc:359] Metadata directory not provided
I20260806 03:02:37.697562 12407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:02:37.698141 12407 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 03:02:37 dist-test-slave-dhph krb5kdc[11814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985357, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.193@KRBTEST.COM for HTTP/127.31.74.212@KRBTEST.COM
I20260806 03:02:37.728154 12407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-0/data/instance:
uuid: "09f2cc3a5d8b4d08830b372a49e32bf7"
format_stamp: "Formatted at 2026-08-06 03:02:37 on dist-test-slave-dhph"
server_key: "c1604b32f85c165f60b26ed8dfebf207"
server_key_iv: "2c2f95d18176a02f52fba816329a2d45"
server_key_version: "kuduclusterkey@0"
I20260806 03:02:37.729461 12407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-0/wal/instance:
uuid: "09f2cc3a5d8b4d08830b372a49e32bf7"
format_stamp: "Formatted at 2026-08-06 03:02:37 on dist-test-slave-dhph"
server_key: "c1604b32f85c165f60b26ed8dfebf207"
server_key_iv: "2c2f95d18176a02f52fba816329a2d45"
server_key_version: "kuduclusterkey@0"
I20260806 03:02:37.738075 12407 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.005s sys 0.000s
I20260806 03:02:37.767407 12428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:02:37.770617 12407 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.008s sys 0.000s
I20260806 03:02:37.770859 12407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-0/wal
uuid: "09f2cc3a5d8b4d08830b372a49e32bf7"
format_stamp: "Formatted at 2026-08-06 03:02:37 on dist-test-slave-dhph"
server_key: "c1604b32f85c165f60b26ed8dfebf207"
server_key_iv: "2c2f95d18176a02f52fba816329a2d45"
server_key_version: "kuduclusterkey@0"
I20260806 03:02:37.771171 12407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:02:37.804073 12407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:02:37.809201 12407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:02:37.809559 12407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:02:37.819058 12407 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:02:37.821365 12407 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:02:37.821498 12407 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:02:37.821640 12407 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:02:37.821722 12407 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:02:37.897650 12407 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.193:36985
I20260806 03:02:37.899518 12407 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
I20260806 03:02:37.897650 12548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.193:36985 every 8 connection(s)
I20260806 03:02:37.902758 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 12407
I20260806 03:02:37.903014 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-0/wal/instance
I20260806 03:02:37.904072 32043 external_mini_cluster.cc:1428] Setting key eb4a6118d2763c754a9844f2f5c1d82d
Aug 06 03:02:37 dist-test-slave-dhph krb5kdc[11814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985357, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.193@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.74.194@KRBTEST.COM; defaulting to no policy
I20260806 03:02:37.961052 12552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:37.908501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:36839 (local address 127.31.74.193:40179)
0806 03:02:37.919616 (+ 11115us) negotiation.cc:107] Waiting for socket to connect
0806 03:02:37.919717 (+ 101us) client_negotiation.cc:174] Beginning negotiation
0806 03:02:37.927638 (+ 7921us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:02:37.933145 (+ 5507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:02:37.933164 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:02:37.934046 (+ 882us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:02:37.935307 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:02:37.935344 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:02:37.938709 (+ 3365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:02:37.938717 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:02:37.941199 (+ 2482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:02:37.941225 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:02:37.943108 (+ 1883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:02:37.943118 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:02:37.943591 (+ 473us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:02:37.944603 (+ 1012us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:02:37.944649 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:02:37.948817 (+ 4168us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:02:37.954756 (+ 5939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:02:37.954771 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:02:37.954798 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:02:37.955742 (+ 944us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:02:37.956381 (+ 639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:02:37.956389 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:02:37.956396 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:02:37.956789 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:02:37.957725 (+ 936us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:02:37.957737 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:02:37.958297 (+ 560us) client_negotiation.cc:769] Sending connection context
0806 03:02:37.958696 (+ 399us) client_negotiation.cc:240] Negotiation successful
0806 03:02:37.959359 (+ 663us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10585,"spinlock_wait_cycles":29056,"thread_start_us":10511,"threads_started":1}
I20260806 03:02:37.962175 12553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:37.909376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.193:40179 (local address 127.31.74.254:36839)
0806 03:02:37.932063 (+ 22687us) server_negotiation.cc:206] Beginning negotiation
0806 03:02:37.932069 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 03:02:37.932098 (+ 29us) server_negotiation.cc:407] Connection header received
0806 03:02:37.932350 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:02:37.932356 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:02:37.932467 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:02:37.932670 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:02:37.937149 (+ 4479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:37.938443 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:37.941484 (+ 3041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:37.942384 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:37.942716 (+ 332us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:02:37.949238 (+ 6522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:02:37.949250 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:02:37.949258 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:02:37.949317 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:02:37.954391 (+ 5074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:02:37.956003 (+ 1612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:02:37.956012 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:02:37.956019 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:02:37.956215 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:02:37.956977 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:02:37.956984 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:02:37.956989 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:02:37.957453 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:02:37.957618 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0806 03:02:37.961487 (+ 3869us) server_negotiation.cc:299] Negotiation successful
0806 03:02:37.961705 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22512,"thread_start_us":22336,"threads_started":1}
I20260806 03:02:37.964915 12549 heartbeater.cc:347] Connected to a master server at 127.31.74.254:36839
I20260806 03:02:37.965736 12549 heartbeater.cc:464] Registering TS with master...
I20260806 03:02:37.967032 12549 heartbeater.cc:511] Master 127.31.74.254:36839 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.194@KRBTEST.COM" created.
I20260806 03:02:37.970217 12322 ts_manager.cc:194] Registered new tserver with Master: 09f2cc3a5d8b4d08830b372a49e32bf7 (127.31.74.193:36985)
I20260806 03:02:37.972559 12322 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.74.193@KRBTEST.COM'} at 127.31.74.193:40179
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.194.keytab.
Entry for principal kudu/127.31.74.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.194.keytab.
W20260806 03:02:38.019917 32043 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.31.74.194: real 0.109s user 0.004s sys 0.028s
I20260806 03:02:38.021538 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.74.212:59733/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.194:0
--local_ip_for_outbound_sockets=127.31.74.194
--webserver_interface=127.31.74.194
--webserver_port=0
--tserver_master_addrs=127.31.74.254:36839
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.194.keytab
--principal=kudu/127.31.74.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:35599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 03:02:38 dist-test-slave-dhph krb5kdc[11814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985357, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.254@KRBTEST.COM for HTTP/127.31.74.212@KRBTEST.COM
Aug 06 03:02:38 dist-test-slave-dhph krb5kdc[11814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985357, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.254@KRBTEST.COM for HTTP/127.31.74.212@KRBTEST.COM
W20260806 03:02:38.581364 12556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:02:38.581905 12556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:02:38.582080 12556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:02:38.594024 12556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:02:38.594184 12556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:02:38.594440 12556 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.194
I20260806 03:02:38.617403 12556 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:35599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.31.74.212:59733/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.194.keytab
--principal=kudu/127.31.74.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.74.194
--webserver_port=0
--tserver_master_addrs=127.31.74.254:36839
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.194
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:02:38.619735 12556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:02:38.622170 12556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:02:38.660458 12563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:02:38.663197 12564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:02:38.677248 12566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:02:38.682981 12556 server_base.cc:1034] running on GCE node
I20260806 03:02:38.684047 12556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:02:38.686251 12556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:02:38.692061 12556 hybrid_clock.cc:648] HybridClock initialized: now 1785985358691925 us; error 109 us; skew 500 ppm
I20260806 03:02:38.692623 12556 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 03:02:38 dist-test-slave-dhph krb5kdc[11814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985358, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:02:38.697999 12556 init.cc:377] Logged in from keytab as kudu/127.31.74.194@KRBTEST.COM (short username kudu)
I20260806 03:02:38.700399 12556 webserver.cc:460] Webserver started at http://127.31.74.194:39943/ using document root <none> and password file <none>
I20260806 03:02:38.701401 12556 fs_manager.cc:359] Metadata directory not provided
I20260806 03:02:38.701565 12556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:02:38.701999 12556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 03:02:38 dist-test-slave-dhph krb5kdc[11814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985358, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.194@KRBTEST.COM for HTTP/127.31.74.212@KRBTEST.COM
I20260806 03:02:38.726634 12556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-1/data/instance:
uuid: "9214eb86a0e44c70a0fe9840386cb199"
format_stamp: "Formatted at 2026-08-06 03:02:38 on dist-test-slave-dhph"
server_key: "459c948972e8acb379d2e59d98a5df7e"
server_key_iv: "35de347c16e8ba0618e4dc52f9eb023d"
server_key_version: "kuduclusterkey@0"
I20260806 03:02:38.728246 12556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-1/wal/instance:
uuid: "9214eb86a0e44c70a0fe9840386cb199"
format_stamp: "Formatted at 2026-08-06 03:02:38 on dist-test-slave-dhph"
server_key: "459c948972e8acb379d2e59d98a5df7e"
server_key_iv: "35de347c16e8ba0618e4dc52f9eb023d"
server_key_version: "kuduclusterkey@0"
I20260806 03:02:38.743763 12556 fs_manager.cc:691] Time spent creating directory manager: real 0.015s user 0.008s sys 0.004s
I20260806 03:02:38.767863 12580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:02:38.769639 12556 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.003s
I20260806 03:02:38.769860 12556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-1/wal
uuid: "9214eb86a0e44c70a0fe9840386cb199"
format_stamp: "Formatted at 2026-08-06 03:02:38 on dist-test-slave-dhph"
server_key: "459c948972e8acb379d2e59d98a5df7e"
server_key_iv: "35de347c16e8ba0618e4dc52f9eb023d"
server_key_version: "kuduclusterkey@0"
I20260806 03:02:38.770112 12556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:02:38.791208 12556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:02:38.796265 12556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:02:38.796634 12556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:02:38.797816 12556 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:02:38.799404 12556 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:02:38.799480 12556 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:02:38.799640 12556 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:02:38.799700 12556 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:02:38.842386 12556 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.194:39009
I20260806 03:02:38.842425 12697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.194:39009 every 8 connection(s)
I20260806 03:02:38.844035 12556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
I20260806 03:02:38.854275 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 12556
I20260806 03:02:38.854492 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-1/wal/instance
I20260806 03:02:38.855175 32043 external_mini_cluster.cc:1428] Setting key 6fb6bea358c2869953f8cfb7b28ff554
Aug 06 03:02:38 dist-test-slave-dhph krb5kdc[11814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985358, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.194@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 03:02:38.881644 12700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:38.848779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.194:59655 (local address 127.31.74.254:36839)
0806 03:02:38.858018 (+ 9239us) server_negotiation.cc:206] Beginning negotiation
0806 03:02:38.858025 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 03:02:38.858047 (+ 22us) server_negotiation.cc:407] Connection header received
0806 03:02:38.858106 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:02:38.858111 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:02:38.858200 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:02:38.858325 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:02:38.860645 (+ 2320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:38.861837 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:38.864270 (+ 2433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:38.865272 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:38.865475 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:02:38.872036 (+ 6561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:02:38.872045 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:02:38.872054 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:02:38.872123 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:02:38.876873 (+ 4750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:02:38.878276 (+ 1403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:02:38.878285 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:02:38.878291 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:02:38.878475 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:02:38.879162 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:02:38.879170 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:02:38.879176 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:02:38.879621 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:02:38.879783 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0806 03:02:38.880822 (+ 1039us) server_negotiation.cc:299] Negotiation successful
0806 03:02:38.881119 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9110,"thread_start_us":8896,"threads_started":1}
I20260806 03:02:38.883082 12701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:38.849531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:36839 (local address 127.31.74.194:59655)
0806 03:02:38.850765 (+ 1234us) negotiation.cc:107] Waiting for socket to connect
0806 03:02:38.850977 (+ 212us) client_negotiation.cc:174] Beginning negotiation
0806 03:02:38.855967 (+ 4990us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:02:38.858773 (+ 2806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:02:38.858788 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:02:38.859480 (+ 692us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:02:38.860389 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:02:38.860425 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:02:38.862049 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:02:38.862054 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:02:38.864059 (+ 2005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:02:38.864089 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:02:38.865590 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:02:38.865598 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:02:38.865946 (+ 348us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:02:38.867281 (+ 1335us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:02:38.867332 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:02:38.871511 (+ 4179us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:02:38.877113 (+ 5602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:02:38.877126 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:02:38.877149 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:02:38.878090 (+ 941us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:02:38.878651 (+ 561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:02:38.878657 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:02:38.878661 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:02:38.878999 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:02:38.879789 (+ 790us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:02:38.879798 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:02:38.880309 (+ 511us) client_negotiation.cc:769] Sending connection context
0806 03:02:38.880656 (+ 347us) client_negotiation.cc:240] Negotiation successful
0806 03:02:38.881368 (+ 712us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":781,"thread_start_us":494,"threads_started":1}
I20260806 03:02:38.884719 12698 heartbeater.cc:347] Connected to a master server at 127.31.74.254:36839
I20260806 03:02:38.885072 12698 heartbeater.cc:464] Registering TS with master...
I20260806 03:02:38.886113 12698 heartbeater.cc:511] Master 127.31.74.254:36839 requested a full tablet report, sending...
I20260806 03:02:38.888209 12322 ts_manager.cc:194] Registered new tserver with Master: 9214eb86a0e44c70a0fe9840386cb199 (127.31.74.194:39009)
I20260806 03:02:38.889222 12322 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.74.194@KRBTEST.COM'} at 127.31.74.194:59655
WARNING: no policy specified for kudu/127.31.74.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.195.keytab.
Entry for principal kudu/127.31.74.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.195.keytab.
I20260806 03:02:38.944810 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.74.212:59733/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.195:0
--local_ip_for_outbound_sockets=127.31.74.195
--webserver_interface=127.31.74.195
--webserver_port=0
--tserver_master_addrs=127.31.74.254:36839
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.195.keytab
--principal=kudu/127.31.74.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:35599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 03:02:38.978034 12549 heartbeater.cc:503] Master 127.31.74.254:36839 was elected leader, sending a full tablet report...
W20260806 03:02:39.340385 12704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:02:39.340713 12704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:02:39.340920 12704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:02:39.350399 12704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:02:39.350556 12704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:02:39.350827 12704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.195
I20260806 03:02:39.361982 12704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:35599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.31.74.212:59733/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.195.keytab
--principal=kudu/127.31.74.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.74.195
--webserver_port=0
--tserver_master_addrs=127.31.74.254:36839
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.195
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:02:39.363691 12704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:02:39.365063 12704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:02:39.380435 12709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:02:39.380683 12712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:02:39.380571 12710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:02:39.382040 12704 server_base.cc:1034] running on GCE node
I20260806 03:02:39.382838 12704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:02:39.384063 12704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:02:39.385540 12704 hybrid_clock.cc:648] HybridClock initialized: now 1785985359385460 us; error 41 us; skew 500 ppm
I20260806 03:02:39.385943 12704 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 03:02:39 dist-test-slave-dhph krb5kdc[11814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985359, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:02:39.389834 12704 init.cc:377] Logged in from keytab as kudu/127.31.74.195@KRBTEST.COM (short username kudu)
I20260806 03:02:39.392004 12704 webserver.cc:460] Webserver started at http://127.31.74.195:37609/ using document root <none> and password file <none>
I20260806 03:02:39.392858 12704 fs_manager.cc:359] Metadata directory not provided
I20260806 03:02:39.393033 12704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:02:39.393395 12704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 03:02:39 dist-test-slave-dhph krb5kdc[11814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985359, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.195@KRBTEST.COM for HTTP/127.31.74.212@KRBTEST.COM
I20260806 03:02:39.411181 12704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-2/data/instance:
uuid: "2598f339060040388868499811a489c6"
format_stamp: "Formatted at 2026-08-06 03:02:39 on dist-test-slave-dhph"
server_key: "0feee16c254ecad7252ab33470f8ef72"
server_key_iv: "c1ec83d26f454a105f16b7c563c5b2a8"
server_key_version: "kuduclusterkey@0"
I20260806 03:02:39.412290 12704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-2/wal/instance:
uuid: "2598f339060040388868499811a489c6"
format_stamp: "Formatted at 2026-08-06 03:02:39 on dist-test-slave-dhph"
server_key: "0feee16c254ecad7252ab33470f8ef72"
server_key_iv: "c1ec83d26f454a105f16b7c563c5b2a8"
server_key_version: "kuduclusterkey@0"
I20260806 03:02:39.418260 12704 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260806 03:02:39.438506 12719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:02:39.440716 12704 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260806 03:02:39.440925 12704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-2/wal
uuid: "2598f339060040388868499811a489c6"
format_stamp: "Formatted at 2026-08-06 03:02:39 on dist-test-slave-dhph"
server_key: "0feee16c254ecad7252ab33470f8ef72"
server_key_iv: "c1ec83d26f454a105f16b7c563c5b2a8"
server_key_version: "kuduclusterkey@0"
I20260806 03:02:39.441195 12704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:02:39.466289 12704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:02:39.471024 12704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:02:39.471407 12704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:02:39.472481 12704 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:02:39.474087 12704 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:02:39.474184 12704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:02:39.474313 12704 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:02:39.474378 12704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:02:39.515826 12704 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.195:46317
I20260806 03:02:39.515980 12834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.195:46317 every 8 connection(s)
I20260806 03:02:39.517418 12704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
I20260806 03:02:39.526497 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 12704
I20260806 03:02:39.526750 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-2/wal/instance
I20260806 03:02:39.527275 32043 external_mini_cluster.cc:1428] Setting key 25c4cb460f64e0fd0f00991e5ad2c558
Aug 06 03:02:39 dist-test-slave-dhph krb5kdc[11814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985359, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.195@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 03:02:39.543133 12837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:39.520378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.195:51925 (local address 127.31.74.254:36839)
0806 03:02:39.520982 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0806 03:02:39.520991 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 03:02:39.522246 (+ 1255us) server_negotiation.cc:407] Connection header received
0806 03:02:39.524424 (+ 2178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:02:39.524429 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:02:39.524506 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:02:39.524612 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:02:39.526349 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:39.527281 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:39.529366 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:39.530082 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:39.530231 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:02:39.534568 (+ 4337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:02:39.534648 (+ 80us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:02:39.534657 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:02:39.534716 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:02:39.538850 (+ 4134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:02:39.539830 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:02:39.539835 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:02:39.539837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:02:39.539978 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:02:39.540554 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:02:39.540558 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:02:39.540560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:02:39.541205 (+ 645us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:02:39.541442 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0806 03:02:39.542527 (+ 1085us) server_negotiation.cc:299] Negotiation successful
0806 03:02:39.542838 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":294,"threads_started":1}
I20260806 03:02:39.544339 12838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:39.521064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:36839 (local address 127.31.74.195:51925)
0806 03:02:39.522045 (+ 981us) negotiation.cc:107] Waiting for socket to connect
0806 03:02:39.522100 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 03:02:39.524001 (+ 1901us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:02:39.524907 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:02:39.524917 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:02:39.525429 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:02:39.526100 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:02:39.526120 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:02:39.527477 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:02:39.527481 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:02:39.529182 (+ 1701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:02:39.529205 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:02:39.530335 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:02:39.530345 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:02:39.530685 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:02:39.531523 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:02:39.531551 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:02:39.534271 (+ 2720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:02:39.539064 (+ 4793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:02:39.539073 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:02:39.539089 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:02:39.539664 (+ 575us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:02:39.540124 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:02:39.540128 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:02:39.540131 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:02:39.540404 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:02:39.541454 (+ 1050us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:02:39.541463 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:02:39.541993 (+ 530us) client_negotiation.cc:769] Sending connection context
0806 03:02:39.542337 (+ 344us) client_negotiation.cc:240] Negotiation successful
0806 03:02:39.543072 (+ 735us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":366,"threads_started":1}
I20260806 03:02:39.546180 12835 heartbeater.cc:347] Connected to a master server at 127.31.74.254:36839
I20260806 03:02:39.546780 12835 heartbeater.cc:464] Registering TS with master...
I20260806 03:02:39.547833 12835 heartbeater.cc:511] Master 127.31.74.254:36839 requested a full tablet report, sending...
I20260806 03:02:39.549669 12322 ts_manager.cc:194] Registered new tserver with Master: 2598f339060040388868499811a489c6 (127.31.74.195:46317)
I20260806 03:02:39.551095 12322 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.74.195@KRBTEST.COM'} at 127.31.74.195:51925
I20260806 03:02:39.558022 32043 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 03:02:39.560331 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 12407
Aug 06 03:02:39 dist-test-slave-dhph krb5kdc[11814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.74.193@KRBTEST.COM
I20260806 03:02:39.582512 12840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:39.561749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55422 (local address 127.31.74.193:36985)
0806 03:02:39.562447 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0806 03:02:39.562455 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 03:02:39.562483 (+ 28us) server_negotiation.cc:407] Connection header received
0806 03:02:39.562550 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:02:39.562557 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:02:39.562949 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:02:39.563108 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:02:39.564842 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:39.566528 (+ 1686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:39.568780 (+ 2252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:39.569490 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:39.569626 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:02:39.575161 (+ 5535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:02:39.575186 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:02:39.575203 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:02:39.575258 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:02:39.578891 (+ 3633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:02:39.579892 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:02:39.579927 (+ 35us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:02:39.579931 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:02:39.580142 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:02:39.580730 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:02:39.580736 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:02:39.580739 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:02:39.581121 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:02:39.581479 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0806 03:02:39.581813 (+ 334us) server_negotiation.cc:299] Negotiation successful
0806 03:02:39.582121 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":355,"threads_started":1}
I20260806 03:02:39.583906 12544 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 03:02:39.678355 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 12407
I20260806 03:02:39.701797 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 12556
Aug 06 03:02:39 dist-test-slave-dhph krb5kdc[11814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.74.194@KRBTEST.COM
I20260806 03:02:39.723567 12842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:39.702960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56616 (local address 127.31.74.194:39009)
0806 03:02:39.703739 (+ 779us) server_negotiation.cc:206] Beginning negotiation
0806 03:02:39.703748 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 03:02:39.703787 (+ 39us) server_negotiation.cc:407] Connection header received
0806 03:02:39.703935 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:02:39.703943 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:02:39.704260 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:02:39.704463 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:02:39.706050 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:39.707599 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:39.709167 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:39.709956 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:39.710094 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:02:39.716261 (+ 6167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:02:39.716278 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:02:39.716303 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:02:39.716362 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:02:39.720026 (+ 3664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:02:39.721048 (+ 1022us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:02:39.721056 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:02:39.721058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:02:39.721230 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:02:39.721727 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:02:39.721731 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:02:39.721733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:02:39.722149 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:02:39.722390 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0806 03:02:39.722965 (+ 575us) server_negotiation.cc:299] Negotiation successful
0806 03:02:39.723230 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":342,"threads_started":1}
I20260806 03:02:39.725538 12693 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 03:02:39.823887 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 12556
I20260806 03:02:39.844101 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 12704
Aug 06 03:02:39 dist-test-slave-dhph krb5kdc[11814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.74.195@KRBTEST.COM
I20260806 03:02:39.868391 12844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:39.845407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56354 (local address 127.31.74.195:46317)
0806 03:02:39.846198 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0806 03:02:39.846208 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 03:02:39.846248 (+ 40us) server_negotiation.cc:407] Connection header received
0806 03:02:39.846313 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:02:39.846320 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:02:39.846716 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:02:39.846908 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:02:39.848498 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:39.849672 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:39.852247 (+ 2575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:39.852886 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:39.853013 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:02:39.859872 (+ 6859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:02:39.859888 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:02:39.859906 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:02:39.859951 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:02:39.863420 (+ 3469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:02:39.864422 (+ 1002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:02:39.864431 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:02:39.864433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:02:39.864608 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:02:39.865387 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:02:39.865392 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:02:39.865394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:02:39.866005 (+ 611us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:02:39.866315 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0806 03:02:39.867444 (+ 1129us) server_negotiation.cc:299] Negotiation successful
0806 03:02:39.867943 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":472,"threads_started":1}
I20260806 03:02:39.870260 12830 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 03:02:39.975862 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 12704
I20260806 03:02:39.996351 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 12289
I20260806 03:02:39.997869 12354 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 03:02:40.126781 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 12289
I20260806 03:02:40.155905 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:36839
--webserver_interface=127.31.74.254
--webserver_port=36475
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab
--principal=kudu/127.31.74.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:35599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.74.254:36839
--ranger_config_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.74.212:59733/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.31.74.212:59733/kms with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:02:40.578675 12847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:02:40.579068 12847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:02:40.579226 12847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:02:40.587694 12847 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:02:40.587881 12847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:02:40.587993 12847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:02:40.588099 12847 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:02:40.588186 12847 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:02:40.604528 12847 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:35599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.31.74.212:59733/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.31.74.254:36839
--ranger_config_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab
--principal=kudu/127.31.74.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:36839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.74.254
--webserver_port=36475
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:02:40.606173 12847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:02:40.607686 12847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:02:40.623219 12852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:02:40.623234 12855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:02:40.623524 12853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:02:40.625056 12847 server_base.cc:1034] running on GCE node
I20260806 03:02:40.626372 12847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:02:40.628275 12847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:02:40.629585 12847 hybrid_clock.cc:648] HybridClock initialized: now 1785985360629507 us; error 85 us; skew 500 ppm
I20260806 03:02:40.630102 12847 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 03:02:40 dist-test-slave-dhph krb5kdc[11814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985360, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:02:40.635417 12847 init.cc:377] Logged in from keytab as kudu/127.31.74.254@KRBTEST.COM (short username kudu)
I20260806 03:02:40.637708 12847 webserver.cc:460] Webserver started at http://127.31.74.254:36475/ using document root <none> and password file <none>
I20260806 03:02:40.638453 12847 fs_manager.cc:359] Metadata directory not provided
I20260806 03:02:40.638682 12847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 03:02:40 dist-test-slave-dhph krb5kdc[11814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985360, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.254@KRBTEST.COM for HTTP/127.31.74.212@KRBTEST.COM
I20260806 03:02:40.725744 12847 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.005s sys 0.000s
I20260806 03:02:40.730509 12863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:02:40.732687 12847 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.002s
I20260806 03:02:40.732908 12847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/master-0/wal
uuid: "60efcc97d32b4aaa968cbb910c59282e"
format_stamp: "Formatted at 2026-08-06 03:02:35 on dist-test-slave-dhph"
server_key: "09e260575ed0a29813ff24f49c0fa370"
server_key_iv: "77a7af5755ec66ece218a251ed9eb13b"
server_key_version: "kuduclusterkey@0"
I20260806 03:02:40.734071 12847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:02:40.757593 12847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:02:40.763065 12847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:02:40.765882 12847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:02:40.792407 12847 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.254:36839
I20260806 03:02:40.792428 12917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.254:36839 every 8 connection(s)
I20260806 03:02:40.794155 12847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
I20260806 03:02:40.796391 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 12847
I20260806 03:02:40.797513 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.74.212:59733/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.193:36985
--local_ip_for_outbound_sockets=127.31.74.193
--tserver_master_addrs=127.31.74.254:36839
--webserver_port=42683
--webserver_interface=127.31.74.193
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.193.keytab
--principal=kudu/127.31.74.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:35599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.31.74.212:59733/kms with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 03:02:40.804992 12918 sys_catalog.cc:263] Verifying existing consensus state
I20260806 03:02:40.810328 12918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60efcc97d32b4aaa968cbb910c59282e: Bootstrap starting.
I20260806 03:02:40.832540 12918 log.cc:824] T 00000000000000000000000000000000 P 60efcc97d32b4aaa968cbb910c59282e: Log is configured to *not* fsync() on all Append() calls
I20260806 03:02:40.845031 12918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60efcc97d32b4aaa968cbb910c59282e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 03:02:40.845705 12918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60efcc97d32b4aaa968cbb910c59282e: Bootstrap complete.
I20260806 03:02:40.857319 12918 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60efcc97d32b4aaa968cbb910c59282e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60efcc97d32b4aaa968cbb910c59282e" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36839 } }
I20260806 03:02:40.858323 12918 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60efcc97d32b4aaa968cbb910c59282e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60efcc97d32b4aaa968cbb910c59282e, State: Initialized, Role: FOLLOWER
I20260806 03:02:40.859220 12918 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60efcc97d32b4aaa968cbb910c59282e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60efcc97d32b4aaa968cbb910c59282e" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36839 } }
I20260806 03:02:40.859493 12918 raft_consensus.cc:388] T 00000000000000000000000000000000 P 60efcc97d32b4aaa968cbb910c59282e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:02:40.859648 12918 raft_consensus.cc:482] T 00000000000000000000000000000000 P 60efcc97d32b4aaa968cbb910c59282e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:02:40.859882 12918 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60efcc97d32b4aaa968cbb910c59282e [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:02:40.864416 12918 raft_consensus.cc:504] T 00000000000000000000000000000000 P 60efcc97d32b4aaa968cbb910c59282e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60efcc97d32b4aaa968cbb910c59282e" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36839 } }
I20260806 03:02:40.865168 12918 leader_election.cc:302] T 00000000000000000000000000000000 P 60efcc97d32b4aaa968cbb910c59282e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 60efcc97d32b4aaa968cbb910c59282e; no voters:
I20260806 03:02:40.866065 12918 leader_election.cc:288] T 00000000000000000000000000000000 P 60efcc97d32b4aaa968cbb910c59282e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 03:02:40.866492 12922 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 60efcc97d32b4aaa968cbb910c59282e [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:02:40.868238 12922 raft_consensus.cc:686] T 00000000000000000000000000000000 P 60efcc97d32b4aaa968cbb910c59282e [term 2 LEADER]: Becoming Leader. State: Replica: 60efcc97d32b4aaa968cbb910c59282e, State: Running, Role: LEADER
I20260806 03:02:40.869199 12918 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60efcc97d32b4aaa968cbb910c59282e [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:02:40.869010 12922 consensus_queue.cc:237] T 00000000000000000000000000000000 P 60efcc97d32b4aaa968cbb910c59282e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60efcc97d32b4aaa968cbb910c59282e" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36839 } }
I20260806 03:02:40.873535 12923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60efcc97d32b4aaa968cbb910c59282e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "60efcc97d32b4aaa968cbb910c59282e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60efcc97d32b4aaa968cbb910c59282e" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36839 } } }
I20260806 03:02:40.874389 12923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60efcc97d32b4aaa968cbb910c59282e [sys.catalog]: This master's current role is: LEADER
I20260806 03:02:40.874286 12924 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60efcc97d32b4aaa968cbb910c59282e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 60efcc97d32b4aaa968cbb910c59282e. Latest consensus state: current_term: 2 leader_uuid: "60efcc97d32b4aaa968cbb910c59282e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60efcc97d32b4aaa968cbb910c59282e" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36839 } } }
I20260806 03:02:40.874861 12924 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60efcc97d32b4aaa968cbb910c59282e [sys.catalog]: This master's current role is: LEADER
I20260806 03:02:40.878301 12928 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:02:40.886965 12928 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:02:40.889482 12928 catalog_manager.cc:1259] Loaded cluster ID: a8bc0021265e47e7915d531336956ccc
I20260806 03:02:40.889787 12928 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:02:40.897655 12928 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:02:40.900807 12928 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 60efcc97d32b4aaa968cbb910c59282e: Loaded TSK: 0
I20260806 03:02:40.903514 12928 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 03:02:40.916250 12918 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260806 03:02:41.424880 12920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:02:41.425467 12920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:02:41.425717 12920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:02:41.442421 12920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:02:41.442633 12920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:02:41.443452 12920 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.193
I20260806 03:02:41.468456 12920 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:35599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.31.74.212:59733/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.193.keytab
--principal=kudu/127.31.74.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.193:36985
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.74.193
--webserver_port=42683
--tserver_master_addrs=127.31.74.254:36839
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.193
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:02:41.470708 12920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:02:41.473227 12920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:02:41.513867 12958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:02:41.516144 12960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:02:41.516443 12957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:02:41.521853 12920 server_base.cc:1034] running on GCE node
I20260806 03:02:41.523097 12920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:02:41.530848 12920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:02:41.540036 12920 hybrid_clock.cc:648] HybridClock initialized: now 1785985361539884 us; error 168 us; skew 500 ppm
I20260806 03:02:41.540864 12920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 03:02:41 dist-test-slave-dhph krb5kdc[11814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985361, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:02:41.546675 12920 init.cc:377] Logged in from keytab as kudu/127.31.74.193@KRBTEST.COM (short username kudu)
I20260806 03:02:41.565210 12920 webserver.cc:460] Webserver started at http://127.31.74.193:42683/ using document root <none> and password file <none>
I20260806 03:02:41.566913 12920 fs_manager.cc:359] Metadata directory not provided
I20260806 03:02:41.567114 12920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 03:02:41 dist-test-slave-dhph krb5kdc[11814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985361, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.193@KRBTEST.COM for HTTP/127.31.74.212@KRBTEST.COM
I20260806 03:02:41.633885 12920 fs_manager.cc:709] Time spent opening directory manager: real 0.018s user 0.010s sys 0.001s
I20260806 03:02:41.653007 12969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:02:41.656733 12920 fs_manager.cc:725] Time spent opening block manager: real 0.019s user 0.001s sys 0.015s
I20260806 03:02:41.657084 12920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-0/wal
uuid: "09f2cc3a5d8b4d08830b372a49e32bf7"
format_stamp: "Formatted at 2026-08-06 03:02:37 on dist-test-slave-dhph"
server_key: "c1604b32f85c165f60b26ed8dfebf207"
server_key_iv: "2c2f95d18176a02f52fba816329a2d45"
server_key_version: "kuduclusterkey@0"
I20260806 03:02:41.659089 12920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:02:41.697657 12920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:02:41.703671 12920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:02:41.704006 12920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:02:41.705554 12920 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:02:41.708665 12920 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:02:41.708840 12920 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:02:41.708961 12920 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:02:41.709044 12920 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:02:41.808141 12920 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.193:36985
I20260806 03:02:41.808187 13087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.193:36985 every 8 connection(s)
I20260806 03:02:41.810405 12920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
I20260806 03:02:41.815989 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 12920
I20260806 03:02:41.817824 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.74.212:59733/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.194:39009
--local_ip_for_outbound_sockets=127.31.74.194
--tserver_master_addrs=127.31.74.254:36839
--webserver_port=39943
--webserver_interface=127.31.74.194
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.194.keytab
--principal=kudu/127.31.74.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:35599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.31.74.212:59733/kms with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 03:02:41 dist-test-slave-dhph krb5kdc[11814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985361, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.193@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 03:02:41.859613 13091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:41.814077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:36839 (local address 127.31.74.193:53227)
0806 03:02:41.819963 (+ 5886us) negotiation.cc:107] Waiting for socket to connect
0806 03:02:41.820026 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0806 03:02:41.822850 (+ 2824us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:02:41.825665 (+ 2815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:02:41.825683 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:02:41.826445 (+ 762us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:02:41.827398 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:02:41.827430 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:02:41.830685 (+ 3255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:02:41.830694 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:02:41.833175 (+ 2481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:02:41.833202 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:02:41.835421 (+ 2219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:02:41.835431 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:02:41.835788 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:02:41.836832 (+ 1044us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:02:41.836884 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:02:41.840647 (+ 3763us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:02:41.852004 (+ 11357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:02:41.852017 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:02:41.852037 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:02:41.853163 (+ 1126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:02:41.853930 (+ 767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:02:41.853936 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:02:41.853940 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:02:41.854432 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:02:41.856174 (+ 1742us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:02:41.856183 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:02:41.856649 (+ 466us) client_negotiation.cc:769] Sending connection context
0806 03:02:41.857113 (+ 464us) client_negotiation.cc:240] Negotiation successful
0806 03:02:41.858035 (+ 922us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5547,"thread_start_us":5280,"threads_started":1}
I20260806 03:02:41.862699 13090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:41.814058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.193:53227 (local address 127.31.74.254:36839)
0806 03:02:41.815238 (+ 1180us) server_negotiation.cc:206] Beginning negotiation
0806 03:02:41.815261 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0806 03:02:41.820225 (+ 4964us) server_negotiation.cc:407] Connection header received
0806 03:02:41.823751 (+ 3526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:02:41.823775 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:02:41.824389 (+ 614us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:02:41.824880 (+ 491us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:02:41.827748 (+ 2868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:41.830330 (+ 2582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:41.833444 (+ 3114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:41.834727 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:41.834948 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:02:41.841149 (+ 6201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:02:41.841162 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:02:41.841187 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:02:41.841263 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:02:41.851677 (+ 10414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:02:41.853428 (+ 1751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:02:41.853440 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:02:41.853450 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:02:41.853738 (+ 288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:02:41.854670 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:02:41.854677 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:02:41.854681 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:02:41.855691 (+ 1010us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:02:41.856070 (+ 379us) server_negotiation.cc:1035] Waiting for connection context
0806 03:02:41.860616 (+ 4546us) server_negotiation.cc:299] Negotiation successful
0806 03:02:41.861432 (+ 816us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":871,"thread_start_us":603,"threads_started":1}
I20260806 03:02:41.868568 13088 heartbeater.cc:347] Connected to a master server at 127.31.74.254:36839
I20260806 03:02:41.868963 13088 heartbeater.cc:464] Registering TS with master...
I20260806 03:02:41.870134 13088 heartbeater.cc:511] Master 127.31.74.254:36839 requested a full tablet report, sending...
W20260806 03:02:41.874423 13088 heartbeater.cc:644] Failed to heartbeat to 127.31.74.254:36839 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 03:02:41.875041 13088 heartbeater.cc:464] Registering TS with master...
I20260806 03:02:41.875550 13088 heartbeater.cc:511] Master 127.31.74.254:36839 requested a full tablet report, sending...
I20260806 03:02:41.877298 13088 heartbeater.cc:464] Registering TS with master...
I20260806 03:02:41.877765 13088 heartbeater.cc:511] Master 127.31.74.254:36839 requested a full tablet report, sending...
W20260806 03:02:41.879320 13088 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 03:02:42.359822 13092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:02:42.360221 13092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:02:42.360426 13092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:02:42.370739 13092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:02:42.371237 13092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:02:42.371598 13092 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.194
I20260806 03:02:42.389410 13092 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:35599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.31.74.212:59733/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.194.keytab
--principal=kudu/127.31.74.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.194:39009
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.74.194
--webserver_port=39943
--tserver_master_addrs=127.31.74.254:36839
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.194
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:02:42.391822 13092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:02:42.394061 13092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:02:42.439296 13108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:02:42.442633 13109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:02:42.443346 13112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:02:42.444007 13092 server_base.cc:1034] running on GCE node
I20260806 03:02:42.445307 13092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:02:42.447633 13092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:02:42.449054 13092 hybrid_clock.cc:648] HybridClock initialized: now 1785985362448993 us; error 56 us; skew 500 ppm
I20260806 03:02:42.449613 13092 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 03:02:42 dist-test-slave-dhph krb5kdc[11814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985362, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:02:42.455508 13092 init.cc:377] Logged in from keytab as kudu/127.31.74.194@KRBTEST.COM (short username kudu)
I20260806 03:02:42.457945 13092 webserver.cc:460] Webserver started at http://127.31.74.194:39943/ using document root <none> and password file <none>
I20260806 03:02:42.459394 13092 fs_manager.cc:359] Metadata directory not provided
I20260806 03:02:42.459596 13092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 03:02:42 dist-test-slave-dhph krb5kdc[11814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985362, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.194@KRBTEST.COM for HTTP/127.31.74.212@KRBTEST.COM
I20260806 03:02:42.490881 13092 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.006s sys 0.000s
I20260806 03:02:42.496025 13121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:02:42.498066 13092 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.001s
I20260806 03:02:42.498287 13092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-1/wal
uuid: "9214eb86a0e44c70a0fe9840386cb199"
format_stamp: "Formatted at 2026-08-06 03:02:38 on dist-test-slave-dhph"
server_key: "459c948972e8acb379d2e59d98a5df7e"
server_key_iv: "35de347c16e8ba0618e4dc52f9eb023d"
server_key_version: "kuduclusterkey@0"
I20260806 03:02:42.499778 13092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Aug 06 03:02:42 dist-test-slave-dhph krb5kdc[11814](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1785985362, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:02:42.531150 13092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:02:42.537263 13092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:02:42.537629 13092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:02:42.539240 13092 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:02:42.544790 13092 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:02:42.545328 13092 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20260806 03:02:42.545547 13092 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:02:42.545647 13092 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:02:42.615415 13092 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.194:39009
I20260806 03:02:42.615456 13237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.194:39009 every 8 connection(s)
I20260806 03:02:42.617595 13092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
I20260806 03:02:42.628100 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 13092
I20260806 03:02:42.629181 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.74.212:59733/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.195:46317
--local_ip_for_outbound_sockets=127.31.74.195
--tserver_master_addrs=127.31.74.254:36839
--webserver_port=37609
--webserver_interface=127.31.74.195
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.195.keytab
--principal=kudu/127.31.74.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:35599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.31.74.212:59733/kms with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 03:02:42 dist-test-slave-dhph krb5kdc[11814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985362, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.194@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 03:02:42.659176 13241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:42.622011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:36839 (local address 127.31.74.194:33269)
0806 03:02:42.624027 (+ 2016us) negotiation.cc:107] Waiting for socket to connect
0806 03:02:42.624083 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0806 03:02:42.627838 (+ 3755us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:02:42.629231 (+ 1393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:02:42.629241 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:02:42.629889 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:02:42.630555 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:02:42.630639 (+ 84us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:02:42.631843 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:02:42.631848 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:02:42.634146 (+ 2298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:02:42.634169 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:02:42.636408 (+ 2239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:02:42.636419 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:02:42.636872 (+ 453us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:02:42.639628 (+ 2756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:02:42.639690 (+ 62us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:02:42.644099 (+ 4409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:02:42.651361 (+ 7262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:02:42.651377 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:02:42.651403 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:02:42.652691 (+ 1288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:02:42.653303 (+ 612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:02:42.653311 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:02:42.653317 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:02:42.654049 (+ 732us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:02:42.655372 (+ 1323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:02:42.655384 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:02:42.655927 (+ 543us) client_negotiation.cc:769] Sending connection context
0806 03:02:42.656513 (+ 586us) client_negotiation.cc:240] Negotiation successful
0806 03:02:42.657293 (+ 780us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1528,"thread_start_us":1270,"threads_started":1}
I20260806 03:02:42.661398 13240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:42.621398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.194:33269 (local address 127.31.74.254:36839)
0806 03:02:42.623730 (+ 2332us) server_negotiation.cc:206] Beginning negotiation
0806 03:02:42.623740 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 03:02:42.624326 (+ 586us) server_negotiation.cc:407] Connection header received
0806 03:02:42.628241 (+ 3915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:02:42.628248 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:02:42.628755 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:02:42.628903 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:02:42.630832 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:42.631629 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:42.634748 (+ 3119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:42.635612 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:42.635781 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:02:42.644521 (+ 8740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:02:42.644530 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:02:42.644536 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:02:42.644597 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:02:42.651082 (+ 6485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:02:42.652939 (+ 1857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:02:42.652949 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:02:42.652956 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:02:42.653149 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:02:42.654298 (+ 1149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:02:42.654307 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:02:42.654314 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:02:42.654917 (+ 603us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:02:42.655270 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0806 03:02:42.659900 (+ 4630us) server_negotiation.cc:299] Negotiation successful
0806 03:02:42.660263 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1874,"thread_start_us":1531,"threads_started":1}
I20260806 03:02:42.665154 13238 heartbeater.cc:347] Connected to a master server at 127.31.74.254:36839
I20260806 03:02:42.665582 13238 heartbeater.cc:464] Registering TS with master...
I20260806 03:02:42.667028 13238 heartbeater.cc:511] Master 127.31.74.254:36839 requested a full tablet report, sending...
I20260806 03:02:42.672472 12881 ts_manager.cc:194] Registered new tserver with Master: 9214eb86a0e44c70a0fe9840386cb199 (127.31.74.194:39009)
I20260806 03:02:42.676343 12881 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.74.194@KRBTEST.COM'} at 127.31.74.194:33269
I20260806 03:02:42.879483 13088 heartbeater.cc:464] Registering TS with master...
I20260806 03:02:42.879983 13088 heartbeater.cc:511] Master 127.31.74.254:36839 requested a full tablet report, sending...
I20260806 03:02:42.882469 12881 ts_manager.cc:194] Registered new tserver with Master: 09f2cc3a5d8b4d08830b372a49e32bf7 (127.31.74.193:36985)
I20260806 03:02:42.888944 12881 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.74.193@KRBTEST.COM'} at 127.31.74.193:53227
Aug 06 03:02:43 dist-test-slave-dhph krb5kdc[11814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985362, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.254@KRBTEST.COM for HTTP/127.31.74.212@KRBTEST.COM
W20260806 03:02:43.237653 13243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:02:43.238499 13243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:02:43.238770 13243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:02:43.254624 13243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:02:43.255060 13243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:02:43.255672 13243 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.195
I20260806 03:02:43.276397 13243 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:35599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.31.74.212:59733/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.195.keytab
--principal=kudu/127.31.74.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.195:46317
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.74.195
--webserver_port=37609
--tserver_master_addrs=127.31.74.254:36839
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.195
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:02:43.278759 13243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:02:43.281100 13243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:02:43.333881 13248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:02:43.334987 13249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:02:43.337064 13243 server_base.cc:1034] running on GCE node
W20260806 03:02:43.339418 13251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:02:43.340704 13243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:02:43.347990 13243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:02:43.352121 13243 hybrid_clock.cc:648] HybridClock initialized: now 1785985363352001 us; error 112 us; skew 500 ppm
I20260806 03:02:43.353122 13243 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 03:02:43 dist-test-slave-dhph krb5kdc[11814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985363, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:02:43.363611 13243 init.cc:377] Logged in from keytab as kudu/127.31.74.195@KRBTEST.COM (short username kudu)
I20260806 03:02:43.367800 13243 webserver.cc:460] Webserver started at http://127.31.74.195:37609/ using document root <none> and password file <none>
I20260806 03:02:43.369220 13243 fs_manager.cc:359] Metadata directory not provided
I20260806 03:02:43.369712 13243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 03:02:43 dist-test-slave-dhph krb5kdc[11814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985363, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.195@KRBTEST.COM for HTTP/127.31.74.212@KRBTEST.COM
I20260806 03:02:43.406632 13243 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.006s sys 0.001s
I20260806 03:02:43.428819 13260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:02:43.433816 13243 fs_manager.cc:725] Time spent opening block manager: real 0.023s user 0.015s sys 0.005s
I20260806 03:02:43.434373 13243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-2/wal
uuid: "2598f339060040388868499811a489c6"
format_stamp: "Formatted at 2026-08-06 03:02:39 on dist-test-slave-dhph"
server_key: "0feee16c254ecad7252ab33470f8ef72"
server_key_iv: "c1ec83d26f454a105f16b7c563c5b2a8"
server_key_version: "kuduclusterkey@0"
I20260806 03:02:43.436272 13243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:02:43.488198 13243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:02:43.495507 13243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:02:43.495939 13243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:02:43.519124 13243 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:02:43.529161 13243 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:02:43.529500 13243 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260806 03:02:43.529723 13243 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:02:43.529848 13243 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:02:43.683738 13238 heartbeater.cc:503] Master 127.31.74.254:36839 was elected leader, sending a full tablet report...
Aug 06 03:02:43 dist-test-slave-dhph krb5kdc[11814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985362, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.254@KRBTEST.COM for HTTP/127.31.74.212@KRBTEST.COM
I20260806 03:02:43.894671 13088 heartbeater.cc:503] Master 127.31.74.254:36839 was elected leader, sending a full tablet report...
I20260806 03:02:43.961534 13243 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.195:46317
I20260806 03:02:43.961618 13382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.195:46317 every 8 connection(s)
I20260806 03:02:43.966488 13243 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
I20260806 03:02:43.971662 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 13243
Aug 06 03:02:44 dist-test-slave-dhph krb5kdc[11814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985363, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.195@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 03:02:44.047632 13384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:43.979069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.195:47661 (local address 127.31.74.254:36839)
0806 03:02:43.999061 (+ 19992us) server_negotiation.cc:206] Beginning negotiation
0806 03:02:43.999067 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 03:02:43.999089 (+ 22us) server_negotiation.cc:407] Connection header received
0806 03:02:44.001079 (+ 1990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:02:44.001086 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:02:44.001189 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:02:44.001323 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:02:44.009798 (+ 8475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:44.011384 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:44.026614 (+ 15230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:44.027458 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:44.027591 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:02:44.033631 (+ 6040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:02:44.033746 (+ 115us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:02:44.033800 (+ 54us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:02:44.033929 (+ 129us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:02:44.039842 (+ 5913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:02:44.041375 (+ 1533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:02:44.041383 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:02:44.041386 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:02:44.041581 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:02:44.042459 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:02:44.042464 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:02:44.042467 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:02:44.042987 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:02:44.043274 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0806 03:02:44.046919 (+ 3645us) server_negotiation.cc:299] Negotiation successful
0806 03:02:44.047205 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":19825,"thread_start_us":15562,"threads_started":1}
I20260806 03:02:44.051249 13389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:43.973349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50948 (local address 127.31.74.254:36839)
0806 03:02:43.992257 (+ 18908us) server_negotiation.cc:206] Beginning negotiation
0806 03:02:43.992266 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 03:02:43.992310 (+ 44us) server_negotiation.cc:407] Connection header received
0806 03:02:43.992418 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:02:43.992424 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:02:43.992529 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:02:43.996735 (+ 4206us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:02:43.997018 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:43.998106 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:44.005993 (+ 7887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:44.006906 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:44.011466 (+ 4560us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:02:44.020034 (+ 8568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:02:44.020043 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:02:44.020048 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:02:44.020128 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:02:44.029648 (+ 9520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:02:44.032563 (+ 2915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:02:44.032570 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:02:44.032575 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:02:44.032942 (+ 367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:02:44.035962 (+ 3020us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:02:44.035968 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:02:44.035973 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:02:44.036894 (+ 921us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:02:44.042471 (+ 5577us) server_negotiation.cc:1035] Waiting for connection context
0806 03:02:44.042705 (+ 234us) server_negotiation.cc:299] Negotiation successful
0806 03:02:44.048462 (+ 5757us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":18676,"thread_start_us":26470,"threads_started":1}
I20260806 03:02:44.057543 13388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:43.978227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:36839 (local address 127.31.74.195:47661)
0806 03:02:43.987781 (+ 9554us) negotiation.cc:107] Waiting for socket to connect
0806 03:02:43.987865 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0806 03:02:44.000667 (+ 12802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:02:44.007532 (+ 6865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:02:44.007548 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:02:44.008399 (+ 851us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:02:44.009451 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:02:44.009486 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:02:44.022890 (+ 13404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:02:44.022902 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:02:44.026241 (+ 3339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:02:44.026273 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:02:44.028099 (+ 1826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:02:44.028108 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:02:44.028442 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:02:44.029463 (+ 1021us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:02:44.029503 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:02:44.033255 (+ 3752us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:02:44.040157 (+ 6902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:02:44.040169 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:02:44.040190 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:02:44.041146 (+ 956us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:02:44.041741 (+ 595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:02:44.041747 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:02:44.041751 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:02:44.042260 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:02:44.043452 (+ 1192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:02:44.043462 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:02:44.044030 (+ 568us) client_negotiation.cc:769] Sending connection context
0806 03:02:44.044383 (+ 353us) client_negotiation.cc:240] Negotiation successful
0806 03:02:44.045470 (+ 1087us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9154,"thread_start_us":8774,"threads_started":1}
I20260806 03:02:44.060468 13383 heartbeater.cc:347] Connected to a master server at 127.31.74.254:36839
I20260806 03:02:44.060851 13383 heartbeater.cc:464] Registering TS with master...
I20260806 03:02:44.061893 13383 heartbeater.cc:511] Master 127.31.74.254:36839 requested a full tablet report, sending...
I20260806 03:02:44.063896 12881 ts_manager.cc:194] Registered new tserver with Master: 2598f339060040388868499811a489c6 (127.31.74.195:46317)
I20260806 03:02:44.065254 12881 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.74.195@KRBTEST.COM'} at 127.31.74.195:47661
I20260806 03:02:44.072957 32043 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 03:02:44.129786 13389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:44.102841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50958 (local address 127.31.74.254:36839)
0806 03:02:44.103098 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0806 03:02:44.103105 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 03:02:44.111546 (+ 8441us) server_negotiation.cc:407] Connection header received
0806 03:02:44.111933 (+ 387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:02:44.111940 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:02:44.112059 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:02:44.112385 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:02:44.116678 (+ 4293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:44.117441 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:44.119082 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:44.119514 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:44.119663 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:02:44.123068 (+ 3405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:02:44.123074 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:02:44.123077 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:02:44.123121 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:02:44.126419 (+ 3298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:02:44.127439 (+ 1020us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:02:44.127468 (+ 29us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:02:44.127470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:02:44.127627 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:02:44.128085 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:02:44.128090 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:02:44.128092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:02:44.128416 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:02:44.128704 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0806 03:02:44.129109 (+ 405us) server_negotiation.cc:299] Negotiation successful
0806 03:02:44.129404 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260806 03:02:44.136354 12881 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50958:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 03:02:44.145399 12881 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 03:02:44.192832 13407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:44.165530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:36985 (local address 127.0.0.1:55424)
0806 03:02:44.178542 (+ 13012us) negotiation.cc:107] Waiting for socket to connect
0806 03:02:44.178630 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0806 03:02:44.178990 (+ 360us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:02:44.181375 (+ 2385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:02:44.181383 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:02:44.181421 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:02:44.181985 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:02:44.182002 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:02:44.184391 (+ 2389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:02:44.184400 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:02:44.187157 (+ 2757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:02:44.187174 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:02:44.190876 (+ 3702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:02:44.190883 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:02:44.191170 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:02:44.191472 (+ 302us) client_negotiation.cc:769] Sending connection context
0806 03:02:44.191662 (+ 190us) client_negotiation.cc:240] Negotiation successful
0806 03:02:44.192242 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12578,"thread_start_us":15928,"threads_started":1}
I20260806 03:02:44.194334 13408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:44.166055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55424 (local address 127.31.74.193:36985)
0806 03:02:44.180083 (+ 14028us) server_negotiation.cc:206] Beginning negotiation
0806 03:02:44.180098 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 03:02:44.180213 (+ 115us) server_negotiation.cc:407] Connection header received
0806 03:02:44.180362 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:02:44.180376 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:02:44.180751 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:02:44.181021 (+ 270us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:02:44.182485 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:44.184188 (+ 1703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:44.187391 (+ 3203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:44.189979 (+ 2588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:44.190168 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:02:44.190661 (+ 493us) server_negotiation.cc:1035] Waiting for connection context
0806 03:02:44.193509 (+ 2848us) server_negotiation.cc:299] Negotiation successful
0806 03:02:44.193825 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13654,"thread_start_us":13456,"threads_started":1}
I20260806 03:02:44.200367 13409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:44.166201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56366 (local address 127.31.74.195:46317)
0806 03:02:44.177116 (+ 10915us) server_negotiation.cc:206] Beginning negotiation
0806 03:02:44.177131 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 03:02:44.181421 (+ 4290us) server_negotiation.cc:407] Connection header received
0806 03:02:44.183645 (+ 2224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:02:44.183657 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:02:44.183926 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:02:44.184095 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:02:44.186792 (+ 2697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:44.189036 (+ 2244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:44.197256 (+ 8220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:44.198917 (+ 1661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:44.199038 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:02:44.199317 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0806 03:02:44.199802 (+ 485us) server_negotiation.cc:299] Negotiation successful
0806 03:02:44.199983 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10616,"thread_start_us":10947,"threads_started":1}
I20260806 03:02:44.200555 13406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:44.165577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.195:46317 (local address 127.0.0.1:56366)
0806 03:02:44.181272 (+ 15695us) negotiation.cc:107] Waiting for socket to connect
0806 03:02:44.181298 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 03:02:44.183410 (+ 2112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:02:44.184476 (+ 1066us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:02:44.184481 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:02:44.184518 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:02:44.185071 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:02:44.185084 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:02:44.192633 (+ 7549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:02:44.192640 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:02:44.196564 (+ 3924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:02:44.196585 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:02:44.199288 (+ 2703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:02:44.199299 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:02:44.199549 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:02:44.199591 (+ 42us) client_negotiation.cc:769] Sending connection context
0806 03:02:44.199698 (+ 107us) client_negotiation.cc:240] Negotiation successful
0806 03:02:44.200048 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15543,"thread_start_us":12285,"threads_started":1}
I20260806 03:02:44.201330 13019 tablet_service.cc:1467] Processing CreateTablet for tablet 95d7c4f334814b2693210459c1a76c53 (DEFAULT_TABLE table=test-table [id=4978380b117d4f37a68ff65d157b4cb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:02:44.204057 13019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95d7c4f334814b2693210459c1a76c53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:02:44.204502 13313 tablet_service.cc:1467] Processing CreateTablet for tablet 95d7c4f334814b2693210459c1a76c53 (DEFAULT_TABLE table=test-table [id=4978380b117d4f37a68ff65d157b4cb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:02:44.206517 13313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95d7c4f334814b2693210459c1a76c53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:02:44.206667 13412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:44.179991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.194:39009 (local address 127.0.0.1:56628)
0806 03:02:44.196360 (+ 16369us) negotiation.cc:107] Waiting for socket to connect
0806 03:02:44.196389 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0806 03:02:44.196558 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:02:44.197654 (+ 1096us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:02:44.197660 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:02:44.197689 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:02:44.198851 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:02:44.198869 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:02:44.201192 (+ 2323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:02:44.201197 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:02:44.203066 (+ 1869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:02:44.203085 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:02:44.205584 (+ 2499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:02:44.205591 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:02:44.205863 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:02:44.205900 (+ 37us) client_negotiation.cc:769] Sending connection context
0806 03:02:44.205999 (+ 99us) client_negotiation.cc:240] Negotiation successful
0806 03:02:44.206180 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16148,"thread_start_us":15971,"threads_started":1}
I20260806 03:02:44.207743 13410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:44.179737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56628 (local address 127.31.74.194:39009)
0806 03:02:44.195406 (+ 15669us) server_negotiation.cc:206] Beginning negotiation
0806 03:02:44.195419 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 03:02:44.196707 (+ 1288us) server_negotiation.cc:407] Connection header received
0806 03:02:44.196856 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:02:44.196865 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:02:44.197180 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:02:44.197412 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:02:44.199073 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:44.200999 (+ 1926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:44.203309 (+ 2310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:44.204980 (+ 1671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:44.205138 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:02:44.205405 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0806 03:02:44.207179 (+ 1774us) server_negotiation.cc:299] Negotiation successful
0806 03:02:44.207397 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15338,"thread_start_us":15700,"threads_started":1}
I20260806 03:02:44.212049 13170 tablet_service.cc:1467] Processing CreateTablet for tablet 95d7c4f334814b2693210459c1a76c53 (DEFAULT_TABLE table=test-table [id=4978380b117d4f37a68ff65d157b4cb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:02:44.214395 13170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95d7c4f334814b2693210459c1a76c53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:02:44.222785 13413 tablet_bootstrap.cc:492] T 95d7c4f334814b2693210459c1a76c53 P 09f2cc3a5d8b4d08830b372a49e32bf7: Bootstrap starting.
I20260806 03:02:44.224192 13414 tablet_bootstrap.cc:492] T 95d7c4f334814b2693210459c1a76c53 P 2598f339060040388868499811a489c6: Bootstrap starting.
I20260806 03:02:44.228809 13413 tablet_bootstrap.cc:654] T 95d7c4f334814b2693210459c1a76c53 P 09f2cc3a5d8b4d08830b372a49e32bf7: Neither blocks nor log segments found. Creating new log.
I20260806 03:02:44.228982 13414 tablet_bootstrap.cc:654] T 95d7c4f334814b2693210459c1a76c53 P 2598f339060040388868499811a489c6: Neither blocks nor log segments found. Creating new log.
I20260806 03:02:44.230062 13413 log.cc:824] T 95d7c4f334814b2693210459c1a76c53 P 09f2cc3a5d8b4d08830b372a49e32bf7: Log is configured to *not* fsync() on all Append() calls
I20260806 03:02:44.230552 13414 log.cc:824] T 95d7c4f334814b2693210459c1a76c53 P 2598f339060040388868499811a489c6: Log is configured to *not* fsync() on all Append() calls
I20260806 03:02:44.233836 13417 tablet_bootstrap.cc:492] T 95d7c4f334814b2693210459c1a76c53 P 9214eb86a0e44c70a0fe9840386cb199: Bootstrap starting.
I20260806 03:02:44.234130 13414 tablet_bootstrap.cc:492] T 95d7c4f334814b2693210459c1a76c53 P 2598f339060040388868499811a489c6: No bootstrap required, opened a new log
I20260806 03:02:44.234198 13413 tablet_bootstrap.cc:492] T 95d7c4f334814b2693210459c1a76c53 P 09f2cc3a5d8b4d08830b372a49e32bf7: No bootstrap required, opened a new log
I20260806 03:02:44.234472 13414 ts_tablet_manager.cc:1397] T 95d7c4f334814b2693210459c1a76c53 P 2598f339060040388868499811a489c6: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260806 03:02:44.234666 13413 ts_tablet_manager.cc:1397] T 95d7c4f334814b2693210459c1a76c53 P 09f2cc3a5d8b4d08830b372a49e32bf7: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260806 03:02:44.237350 13417 tablet_bootstrap.cc:654] T 95d7c4f334814b2693210459c1a76c53 P 9214eb86a0e44c70a0fe9840386cb199: Neither blocks nor log segments found. Creating new log.
I20260806 03:02:44.238451 13417 log.cc:824] T 95d7c4f334814b2693210459c1a76c53 P 9214eb86a0e44c70a0fe9840386cb199: Log is configured to *not* fsync() on all Append() calls
I20260806 03:02:44.241837 13417 tablet_bootstrap.cc:492] T 95d7c4f334814b2693210459c1a76c53 P 9214eb86a0e44c70a0fe9840386cb199: No bootstrap required, opened a new log
I20260806 03:02:44.242353 13417 ts_tablet_manager.cc:1397] T 95d7c4f334814b2693210459c1a76c53 P 9214eb86a0e44c70a0fe9840386cb199: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260806 03:02:44.246505 13414 raft_consensus.cc:348] T 95d7c4f334814b2693210459c1a76c53 P 2598f339060040388868499811a489c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2598f339060040388868499811a489c6" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 46317 } } peers { permanent_uuid: "09f2cc3a5d8b4d08830b372a49e32bf7" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 36985 } } peers { permanent_uuid: "9214eb86a0e44c70a0fe9840386cb199" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 39009 } }
I20260806 03:02:44.247066 13414 raft_consensus.cc:374] T 95d7c4f334814b2693210459c1a76c53 P 2598f339060040388868499811a489c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:02:44.246505 13413 raft_consensus.cc:348] T 95d7c4f334814b2693210459c1a76c53 P 09f2cc3a5d8b4d08830b372a49e32bf7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2598f339060040388868499811a489c6" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 46317 } } peers { permanent_uuid: "09f2cc3a5d8b4d08830b372a49e32bf7" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 36985 } } peers { permanent_uuid: "9214eb86a0e44c70a0fe9840386cb199" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 39009 } }
I20260806 03:02:44.247179 13414 raft_consensus.cc:729] T 95d7c4f334814b2693210459c1a76c53 P 2598f339060040388868499811a489c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2598f339060040388868499811a489c6, State: Initialized, Role: FOLLOWER
I20260806 03:02:44.247238 13413 raft_consensus.cc:374] T 95d7c4f334814b2693210459c1a76c53 P 09f2cc3a5d8b4d08830b372a49e32bf7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:02:44.247341 13413 raft_consensus.cc:729] T 95d7c4f334814b2693210459c1a76c53 P 09f2cc3a5d8b4d08830b372a49e32bf7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09f2cc3a5d8b4d08830b372a49e32bf7, State: Initialized, Role: FOLLOWER
I20260806 03:02:44.247999 13414 consensus_queue.cc:260] T 95d7c4f334814b2693210459c1a76c53 P 2598f339060040388868499811a489c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2598f339060040388868499811a489c6" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 46317 } } peers { permanent_uuid: "09f2cc3a5d8b4d08830b372a49e32bf7" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 36985 } } peers { permanent_uuid: "9214eb86a0e44c70a0fe9840386cb199" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 39009 } }
I20260806 03:02:44.247999 13413 consensus_queue.cc:260] T 95d7c4f334814b2693210459c1a76c53 P 09f2cc3a5d8b4d08830b372a49e32bf7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2598f339060040388868499811a489c6" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 46317 } } peers { permanent_uuid: "09f2cc3a5d8b4d08830b372a49e32bf7" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 36985 } } peers { permanent_uuid: "9214eb86a0e44c70a0fe9840386cb199" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 39009 } }
I20260806 03:02:44.252287 13383 heartbeater.cc:503] Master 127.31.74.254:36839 was elected leader, sending a full tablet report...
I20260806 03:02:44.252437 13413 ts_tablet_manager.cc:1428] T 95d7c4f334814b2693210459c1a76c53 P 09f2cc3a5d8b4d08830b372a49e32bf7: Time spent starting tablet: real 0.018s user 0.015s sys 0.000s
I20260806 03:02:44.252458 13414 ts_tablet_manager.cc:1428] T 95d7c4f334814b2693210459c1a76c53 P 2598f339060040388868499811a489c6: Time spent starting tablet: real 0.018s user 0.013s sys 0.005s
I20260806 03:02:44.257270 13417 raft_consensus.cc:348] T 95d7c4f334814b2693210459c1a76c53 P 9214eb86a0e44c70a0fe9840386cb199 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2598f339060040388868499811a489c6" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 46317 } } peers { permanent_uuid: "09f2cc3a5d8b4d08830b372a49e32bf7" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 36985 } } peers { permanent_uuid: "9214eb86a0e44c70a0fe9840386cb199" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 39009 } }
I20260806 03:02:44.258019 13417 raft_consensus.cc:374] T 95d7c4f334814b2693210459c1a76c53 P 9214eb86a0e44c70a0fe9840386cb199 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:02:44.258167 13417 raft_consensus.cc:729] T 95d7c4f334814b2693210459c1a76c53 P 9214eb86a0e44c70a0fe9840386cb199 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9214eb86a0e44c70a0fe9840386cb199, State: Initialized, Role: FOLLOWER
I20260806 03:02:44.259050 13417 consensus_queue.cc:260] T 95d7c4f334814b2693210459c1a76c53 P 9214eb86a0e44c70a0fe9840386cb199 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2598f339060040388868499811a489c6" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 46317 } } peers { permanent_uuid: "09f2cc3a5d8b4d08830b372a49e32bf7" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 36985 } } peers { permanent_uuid: "9214eb86a0e44c70a0fe9840386cb199" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 39009 } }
I20260806 03:02:44.261248 13417 ts_tablet_manager.cc:1428] T 95d7c4f334814b2693210459c1a76c53 P 9214eb86a0e44c70a0fe9840386cb199: Time spent starting tablet: real 0.019s user 0.011s sys 0.007s
W20260806 03:02:44.315296 13089 tablet.cc:2369] T 95d7c4f334814b2693210459c1a76c53 P 09f2cc3a5d8b4d08830b372a49e32bf7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 03:02:44.371830 13239 tablet.cc:2369] T 95d7c4f334814b2693210459c1a76c53 P 9214eb86a0e44c70a0fe9840386cb199: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 03:02:44.382869 13420 raft_consensus.cc:482] T 95d7c4f334814b2693210459c1a76c53 P 2598f339060040388868499811a489c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:02:44.383440 13420 raft_consensus.cc:504] T 95d7c4f334814b2693210459c1a76c53 P 2598f339060040388868499811a489c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2598f339060040388868499811a489c6" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 46317 } } peers { permanent_uuid: "09f2cc3a5d8b4d08830b372a49e32bf7" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 36985 } } peers { permanent_uuid: "9214eb86a0e44c70a0fe9840386cb199" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 39009 } }
I20260806 03:02:44.385649 13420 leader_election.cc:288] T 95d7c4f334814b2693210459c1a76c53 P 2598f339060040388868499811a489c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09f2cc3a5d8b4d08830b372a49e32bf7 (127.31.74.193:36985), 9214eb86a0e44c70a0fe9840386cb199 (127.31.74.194:39009)
I20260806 03:02:44.393482 13388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:44.386083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.194:39009 (local address 127.31.74.195:54397)
0806 03:02:44.386241 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0806 03:02:44.386366 (+ 125us) client_negotiation.cc:174] Beginning negotiation
0806 03:02:44.386661 (+ 295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:02:44.387268 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:02:44.387275 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:02:44.387315 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:02:44.387567 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:02:44.387582 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:02:44.388822 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:02:44.388827 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:02:44.391092 (+ 2265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:02:44.391122 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:02:44.392854 (+ 1732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:02:44.392859 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:02:44.393011 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:02:44.393038 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 03:02:44.393129 (+ 91us) client_negotiation.cc:240] Negotiation successful
0806 03:02:44.393234 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260806 03:02:44.394364 13422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:44.386134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:36985 (local address 127.31.74.195:52003)
0806 03:02:44.387458 (+ 1324us) negotiation.cc:107] Waiting for socket to connect
0806 03:02:44.387488 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0806 03:02:44.387650 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:02:44.388137 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:02:44.388141 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:02:44.388168 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:02:44.388502 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:02:44.388512 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:02:44.389705 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:02:44.389713 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:02:44.391882 (+ 2169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:02:44.391895 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:02:44.393715 (+ 1820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:02:44.393721 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:02:44.393923 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:02:44.393950 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 03:02:44.394027 (+ 77us) client_negotiation.cc:240] Negotiation successful
0806 03:02:44.394122 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1180,"thread_start_us":855,"threads_started":1}
I20260806 03:02:44.394404 13410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:44.386115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.195:54397 (local address 127.31.74.194:39009)
0806 03:02:44.386388 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0806 03:02:44.386394 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 03:02:44.386604 (+ 210us) server_negotiation.cc:407] Connection header received
0806 03:02:44.386875 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:02:44.386884 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:02:44.387001 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:02:44.387120 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:02:44.387766 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:44.388661 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:44.391453 (+ 2792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:44.392640 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:44.392837 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:02:44.392929 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0806 03:02:44.393197 (+ 268us) server_negotiation.cc:299] Negotiation successful
0806 03:02:44.393369 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260806 03:02:44.394621 13408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:44.385826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.195:52003 (local address 127.31.74.193:36985)
0806 03:02:44.386073 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0806 03:02:44.386080 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 03:02:44.387622 (+ 1542us) server_negotiation.cc:407] Connection header received
0806 03:02:44.387823 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:02:44.387831 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:02:44.387924 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:02:44.388035 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:02:44.388668 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:44.389397 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:44.392051 (+ 2654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:44.393475 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:44.393659 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:02:44.393746 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 03:02:44.394099 (+ 353us) server_negotiation.cc:299] Negotiation successful
0806 03:02:44.394305 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260806 03:02:44.395573 13193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95d7c4f334814b2693210459c1a76c53" candidate_uuid: "2598f339060040388868499811a489c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9214eb86a0e44c70a0fe9840386cb199" is_pre_election: true
I20260806 03:02:44.395604 13040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95d7c4f334814b2693210459c1a76c53" candidate_uuid: "2598f339060040388868499811a489c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09f2cc3a5d8b4d08830b372a49e32bf7" is_pre_election: true
I20260806 03:02:44.396183 13193 raft_consensus.cc:2454] T 95d7c4f334814b2693210459c1a76c53 P 9214eb86a0e44c70a0fe9840386cb199 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2598f339060040388868499811a489c6 in term 0.
I20260806 03:02:44.396186 13040 raft_consensus.cc:2454] T 95d7c4f334814b2693210459c1a76c53 P 09f2cc3a5d8b4d08830b372a49e32bf7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2598f339060040388868499811a489c6 in term 0.
I20260806 03:02:44.396981 13261 leader_election.cc:302] T 95d7c4f334814b2693210459c1a76c53 P 2598f339060040388868499811a489c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2598f339060040388868499811a489c6, 9214eb86a0e44c70a0fe9840386cb199; no voters:
I20260806 03:02:44.397501 13420 raft_consensus.cc:2781] T 95d7c4f334814b2693210459c1a76c53 P 2598f339060040388868499811a489c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:02:44.397667 13420 raft_consensus.cc:482] T 95d7c4f334814b2693210459c1a76c53 P 2598f339060040388868499811a489c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:02:44.397779 13420 raft_consensus.cc:3037] T 95d7c4f334814b2693210459c1a76c53 P 2598f339060040388868499811a489c6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:02:44.400589 13420 raft_consensus.cc:504] T 95d7c4f334814b2693210459c1a76c53 P 2598f339060040388868499811a489c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2598f339060040388868499811a489c6" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 46317 } } peers { permanent_uuid: "09f2cc3a5d8b4d08830b372a49e32bf7" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 36985 } } peers { permanent_uuid: "9214eb86a0e44c70a0fe9840386cb199" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 39009 } }
I20260806 03:02:44.401623 13420 leader_election.cc:288] T 95d7c4f334814b2693210459c1a76c53 P 2598f339060040388868499811a489c6 [CANDIDATE]: Term 1 election: Requested vote from peers 09f2cc3a5d8b4d08830b372a49e32bf7 (127.31.74.193:36985), 9214eb86a0e44c70a0fe9840386cb199 (127.31.74.194:39009)
I20260806 03:02:44.401938 13040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95d7c4f334814b2693210459c1a76c53" candidate_uuid: "2598f339060040388868499811a489c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09f2cc3a5d8b4d08830b372a49e32bf7"
I20260806 03:02:44.402184 13040 raft_consensus.cc:3037] T 95d7c4f334814b2693210459c1a76c53 P 09f2cc3a5d8b4d08830b372a49e32bf7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:02:44.402273 13193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95d7c4f334814b2693210459c1a76c53" candidate_uuid: "2598f339060040388868499811a489c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9214eb86a0e44c70a0fe9840386cb199"
I20260806 03:02:44.402547 13193 raft_consensus.cc:3037] T 95d7c4f334814b2693210459c1a76c53 P 9214eb86a0e44c70a0fe9840386cb199 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:02:44.404762 13040 raft_consensus.cc:2454] T 95d7c4f334814b2693210459c1a76c53 P 09f2cc3a5d8b4d08830b372a49e32bf7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2598f339060040388868499811a489c6 in term 1.
I20260806 03:02:44.405658 13193 raft_consensus.cc:2454] T 95d7c4f334814b2693210459c1a76c53 P 9214eb86a0e44c70a0fe9840386cb199 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2598f339060040388868499811a489c6 in term 1.
I20260806 03:02:44.405861 13264 leader_election.cc:302] T 95d7c4f334814b2693210459c1a76c53 P 2598f339060040388868499811a489c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09f2cc3a5d8b4d08830b372a49e32bf7, 2598f339060040388868499811a489c6; no voters:
I20260806 03:02:44.406399 13420 raft_consensus.cc:2781] T 95d7c4f334814b2693210459c1a76c53 P 2598f339060040388868499811a489c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:02:44.407064 13420 raft_consensus.cc:686] T 95d7c4f334814b2693210459c1a76c53 P 2598f339060040388868499811a489c6 [term 1 LEADER]: Becoming Leader. State: Replica: 2598f339060040388868499811a489c6, State: Running, Role: LEADER
I20260806 03:02:44.407662 13420 consensus_queue.cc:237] T 95d7c4f334814b2693210459c1a76c53 P 2598f339060040388868499811a489c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2598f339060040388868499811a489c6" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 46317 } } peers { permanent_uuid: "09f2cc3a5d8b4d08830b372a49e32bf7" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 36985 } } peers { permanent_uuid: "9214eb86a0e44c70a0fe9840386cb199" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 39009 } }
I20260806 03:02:44.415738 12880 catalog_manager.cc:5495] T 95d7c4f334814b2693210459c1a76c53 P 2598f339060040388868499811a489c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2598f339060040388868499811a489c6 (127.31.74.195). New cstate: current_term: 1 leader_uuid: "2598f339060040388868499811a489c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2598f339060040388868499811a489c6" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 46317 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09f2cc3a5d8b4d08830b372a49e32bf7" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 36985 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9214eb86a0e44c70a0fe9840386cb199" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 39009 } health_report { overall_health: UNKNOWN } } }
I20260806 03:02:44.452286 13409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:44.443424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56382 (local address 127.31.74.195:46317)
0806 03:02:44.443780 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0806 03:02:44.443787 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 03:02:44.443817 (+ 30us) server_negotiation.cc:407] Connection header received
0806 03:02:44.444135 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:02:44.444142 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:02:44.444244 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:02:44.444441 (+ 197us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 03:02:44.445047 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:44.445926 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:44.448624 (+ 2698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:44.449391 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:44.449588 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:02:44.450261 (+ 673us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 03:02:44.451363 (+ 1102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 03:02:44.451552 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0806 03:02:44.451643 (+ 91us) server_negotiation.cc:299] Negotiation successful
0806 03:02:44.451812 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260806 03:02:44.463346 13193 raft_consensus.cc:1261] T 95d7c4f334814b2693210459c1a76c53 P 9214eb86a0e44c70a0fe9840386cb199 [term 1 FOLLOWER]: Refusing update from remote peer 2598f339060040388868499811a489c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:02:44.463341 13040 raft_consensus.cc:1261] T 95d7c4f334814b2693210459c1a76c53 P 09f2cc3a5d8b4d08830b372a49e32bf7 [term 1 FOLLOWER]: Refusing update from remote peer 2598f339060040388868499811a489c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:02:44.464929 13420 consensus_queue.cc:1035] T 95d7c4f334814b2693210459c1a76c53 P 2598f339060040388868499811a489c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9214eb86a0e44c70a0fe9840386cb199" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 39009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:02:44.465518 13423 consensus_queue.cc:1035] T 95d7c4f334814b2693210459c1a76c53 P 2598f339060040388868499811a489c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09f2cc3a5d8b4d08830b372a49e32bf7" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 36985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:02:44.501585 12880 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50958:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 03:02:44.501924 12880 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50958:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 03:02:44.507725 12880 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 60efcc97d32b4aaa968cbb910c59282e: Sending DeleteTablet for 3 replicas of tablet 95d7c4f334814b2693210459c1a76c53
I20260806 03:02:44.509517 13019 tablet_service.cc:1514] Processing DeleteTablet for tablet 95d7c4f334814b2693210459c1a76c53 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:02:44 UTC) from {username='kudu', principal='kudu/127.31.74.254@KRBTEST.COM'} at 127.0.0.1:55424
I20260806 03:02:44.510371 13170 tablet_service.cc:1514] Processing DeleteTablet for tablet 95d7c4f334814b2693210459c1a76c53 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:02:44 UTC) from {username='kudu', principal='kudu/127.31.74.254@KRBTEST.COM'} at 127.0.0.1:56628
I20260806 03:02:44.510975 13313 tablet_service.cc:1514] Processing DeleteTablet for tablet 95d7c4f334814b2693210459c1a76c53 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:02:44 UTC) from {username='kudu', principal='kudu/127.31.74.254@KRBTEST.COM'} at 127.0.0.1:56366
I20260806 03:02:44.520406 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 12920
I20260806 03:02:44.526450 13437 tablet_replica.cc:330] T 95d7c4f334814b2693210459c1a76c53 P 09f2cc3a5d8b4d08830b372a49e32bf7: stopping tablet replica
I20260806 03:02:44.526520 13438 tablet_replica.cc:330] T 95d7c4f334814b2693210459c1a76c53 P 9214eb86a0e44c70a0fe9840386cb199: stopping tablet replica
I20260806 03:02:44.527469 13438 raft_consensus.cc:2229] T 95d7c4f334814b2693210459c1a76c53 P 9214eb86a0e44c70a0fe9840386cb199 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:02:44.528465 13438 raft_consensus.cc:2258] T 95d7c4f334814b2693210459c1a76c53 P 9214eb86a0e44c70a0fe9840386cb199 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:02:44.530397 13437 raft_consensus.cc:2229] T 95d7c4f334814b2693210459c1a76c53 P 09f2cc3a5d8b4d08830b372a49e32bf7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:02:44.531872 13437 raft_consensus.cc:2258] T 95d7c4f334814b2693210459c1a76c53 P 09f2cc3a5d8b4d08830b372a49e32bf7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:02:44.533808 13438 ts_tablet_manager.cc:1905] T 95d7c4f334814b2693210459c1a76c53 P 9214eb86a0e44c70a0fe9840386cb199: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:02:44.536813 13437 ts_tablet_manager.cc:1905] T 95d7c4f334814b2693210459c1a76c53 P 09f2cc3a5d8b4d08830b372a49e32bf7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:02:44.539837 13439 tablet_replica.cc:330] T 95d7c4f334814b2693210459c1a76c53 P 2598f339060040388868499811a489c6: stopping tablet replica
I20260806 03:02:44.540480 13439 raft_consensus.cc:2229] T 95d7c4f334814b2693210459c1a76c53 P 2598f339060040388868499811a489c6 [term 1 LEADER]: Raft consensus shutting down.
I20260806 03:02:44.541304 13439 raft_consensus.cc:2258] T 95d7c4f334814b2693210459c1a76c53 P 2598f339060040388868499811a489c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:02:44.542342 13438 ts_tablet_manager.cc:1918] T 95d7c4f334814b2693210459c1a76c53 P 9214eb86a0e44c70a0fe9840386cb199: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 03:02:44.542526 13438 log.cc:1196] T 95d7c4f334814b2693210459c1a76c53 P 9214eb86a0e44c70a0fe9840386cb199: Deleting WAL directory at /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-1/wal/wals/95d7c4f334814b2693210459c1a76c53
I20260806 03:02:44.543150 13438 ts_tablet_manager.cc:1939] T 95d7c4f334814b2693210459c1a76c53 P 9214eb86a0e44c70a0fe9840386cb199: Deleting consensus metadata
I20260806 03:02:44.544284 13439 ts_tablet_manager.cc:1905] T 95d7c4f334814b2693210459c1a76c53 P 2598f339060040388868499811a489c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:02:44.545173 12864 catalog_manager.cc:4841] TS 9214eb86a0e44c70a0fe9840386cb199 (127.31.74.194:39009): tablet 95d7c4f334814b2693210459c1a76c53 (table test-table [id=4978380b117d4f37a68ff65d157b4cb7]) successfully deleted
I20260806 03:02:44.546990 13437 ts_tablet_manager.cc:1918] T 95d7c4f334814b2693210459c1a76c53 P 09f2cc3a5d8b4d08830b372a49e32bf7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 03:02:44.547582 13437 log.cc:1196] T 95d7c4f334814b2693210459c1a76c53 P 09f2cc3a5d8b4d08830b372a49e32bf7: Deleting WAL directory at /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-0/wal/wals/95d7c4f334814b2693210459c1a76c53
I20260806 03:02:44.549152 13408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:44.521577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55428 (local address 127.31.74.193:36985)
0806 03:02:44.521764 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0806 03:02:44.521774 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 03:02:44.521922 (+ 148us) server_negotiation.cc:407] Connection header received
0806 03:02:44.522190 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:02:44.522198 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:02:44.522285 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:02:44.522426 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:02:44.524957 (+ 2531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:44.526522 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:44.530060 (+ 3538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:44.532014 (+ 1954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:44.532210 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:02:44.537584 (+ 5374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:02:44.537600 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:02:44.537619 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:02:44.537672 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:02:44.542534 (+ 4862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:02:44.543814 (+ 1280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:02:44.543825 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:02:44.543828 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:02:44.544018 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:02:44.544552 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:02:44.544557 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:02:44.544560 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:02:44.545091 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:02:44.545500 (+ 409us) server_negotiation.cc:1035] Waiting for connection context
0806 03:02:44.548086 (+ 2586us) server_negotiation.cc:299] Negotiation successful
0806 03:02:44.548508 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 03:02:44.548867 13437 ts_tablet_manager.cc:1939] T 95d7c4f334814b2693210459c1a76c53 P 09f2cc3a5d8b4d08830b372a49e32bf7: Deleting consensus metadata
I20260806 03:02:44.551782 13083 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 03:02:44.553234 12867 catalog_manager.cc:4841] TS 09f2cc3a5d8b4d08830b372a49e32bf7 (127.31.74.193:36985): tablet 95d7c4f334814b2693210459c1a76c53 (table test-table [id=4978380b117d4f37a68ff65d157b4cb7]) successfully deleted
I20260806 03:02:44.555958 13439 ts_tablet_manager.cc:1918] T 95d7c4f334814b2693210459c1a76c53 P 2598f339060040388868499811a489c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 03:02:44.556216 13439 log.cc:1196] T 95d7c4f334814b2693210459c1a76c53 P 2598f339060040388868499811a489c6: Deleting WAL directory at /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785985105255475-32043-0/minicluster-data/ts-2/wal/wals/95d7c4f334814b2693210459c1a76c53
I20260806 03:02:44.557374 13439 ts_tablet_manager.cc:1939] T 95d7c4f334814b2693210459c1a76c53 P 2598f339060040388868499811a489c6: Deleting consensus metadata
I20260806 03:02:44.559901 12864 catalog_manager.cc:4841] TS 2598f339060040388868499811a489c6 (127.31.74.195:46317): tablet 95d7c4f334814b2693210459c1a76c53 (table test-table [id=4978380b117d4f37a68ff65d157b4cb7]) successfully deleted
I20260806 03:02:44.684203 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 12920
I20260806 03:02:44.719465 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 13092
I20260806 03:02:44.740703 13410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:44.720642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56636 (local address 127.31.74.194:39009)
0806 03:02:44.720857 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0806 03:02:44.720864 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 03:02:44.720986 (+ 122us) server_negotiation.cc:407] Connection header received
0806 03:02:44.721195 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:02:44.721202 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:02:44.721323 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:02:44.721472 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:02:44.723778 (+ 2306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:44.724964 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:44.726538 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:44.727225 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:44.727377 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:02:44.730498 (+ 3121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:02:44.730522 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:02:44.730545 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:02:44.730698 (+ 153us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:02:44.735834 (+ 5136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:02:44.736810 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:02:44.736820 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:02:44.736824 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:02:44.737017 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:02:44.737552 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:02:44.737557 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:02:44.737561 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:02:44.738256 (+ 695us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:02:44.738794 (+ 538us) server_negotiation.cc:1035] Waiting for connection context
0806 03:02:44.739861 (+ 1067us) server_negotiation.cc:299] Negotiation successful
0806 03:02:44.740180 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260806 03:02:44.741695 13233 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 03:02:44.849323 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 13092
I20260806 03:02:44.875834 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 13243
I20260806 03:02:44.897701 13409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:44.876943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56396 (local address 127.31.74.195:46317)
0806 03:02:44.877196 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0806 03:02:44.877204 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 03:02:44.877235 (+ 31us) server_negotiation.cc:407] Connection header received
0806 03:02:44.877441 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:02:44.877450 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:02:44.877554 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:02:44.877686 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:02:44.879417 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:44.881222 (+ 1805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:44.882922 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:44.883651 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:44.883833 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:02:44.887759 (+ 3926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:02:44.887778 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:02:44.887801 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:02:44.887884 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:02:44.893251 (+ 5367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:02:44.894413 (+ 1162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:02:44.894424 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:02:44.894430 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:02:44.894702 (+ 272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:02:44.895291 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:02:44.895297 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:02:44.895300 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:02:44.895804 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:02:44.896472 (+ 668us) server_negotiation.cc:1035] Waiting for connection context
0806 03:02:44.896817 (+ 345us) server_negotiation.cc:299] Negotiation successful
0806 03:02:44.897177 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260806 03:02:44.899052 13378 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 03:02:45.051743 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 13243
I20260806 03:02:45.077430 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 12847
I20260806 03:02:45.079114 12913 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 03:02:45.225553 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 12847
I20260806 03:02:45.255555 32043 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260806 03:02:45.603348 32043 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260806 03:02:45.603716 32043 mini_ranger.cc:66] Stopping Ranger...
I20260806 03:02:45.964668 32043 mini_ranger.cc:68] Stopped Ranger
2026-08-06 03:02:45.965 UTC [11848] LOG: received smart shutdown request
2026-08-06 03:02:45.970 UTC [11848] LOG: background worker "logical replication launcher" (PID 11857) exited with exit code 1
2026-08-06 03:02:45.971 UTC [11852] LOG: shutting down
2026-08-06 03:02:46.272 UTC [11848] LOG: database system is shut down
2026-08-06T03:02:46Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (72054 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785985105255475-32043-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 03:02:46 dist-test-slave-dhph krb5kdc[13453](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 03:02:46 dist-test-slave-dhph krb5kdc[13453](info): set up 2 sockets
Aug 06 03:02:46 dist-test-slave-dhph krb5kdc[13453](info): commencing operation
krb5kdc: starting...
W20260806 03:02:47.774915 32043 mini_kdc.cc:121] Time spent starting KDC: real 1.270s user 0.003s sys 0.047s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785985105255475-32043-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785985105255475-32043-0/krb5kdc/test-user.keytab.
Aug 06 03:02:47 dist-test-slave-dhph krb5kdc[13453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985367, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T03:02:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:02:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.74.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab.
Entry for principal kudu/127.31.74.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab.
I20260806 03:02:48.123785 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785985105255475-32043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785985105255475-32043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785985105255475-32043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:38641
--webserver_interface=127.31.74.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab
--principal=kudu/127.31.74.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:33253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.74.254:38641,127.31.74.253:37437,127.31.74.252:34107
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:02:48.513074 13467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:02:48.513407 13467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:02:48.513563 13467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:02:48.521662 13467 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:02:48.521817 13467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:02:48.521908 13467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:02:48.522012 13467 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:02:48.522110 13467 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:02:48.533473 13467 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:33253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785985105255475-32043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785985105255475-32043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.74.254:38641,127.31.74.253:37437,127.31.74.252:34107
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab
--principal=kudu/127.31.74.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:38641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.74.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785985105255475-32043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:02:48.535825 13467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:02:48.537914 13467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:02:48.557473 13473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:02:48.557885 13475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:02:48.557973 13472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:02:48.558561 13467 server_base.cc:1034] running on GCE node
I20260806 03:02:48.559533 13467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:02:48.561264 13467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:02:48.562541 13467 hybrid_clock.cc:648] HybridClock initialized: now 1785985368562491 us; error 42 us; skew 500 ppm
I20260806 03:02:48.563019 13467 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 03:02:48 dist-test-slave-dhph krb5kdc[13453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985368, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:02:48.566809 13467 init.cc:377] Logged in from keytab as kudu/127.31.74.254@KRBTEST.COM (short username kudu)
I20260806 03:02:48.568696 13467 webserver.cc:460] Webserver started at http://127.31.74.254:39145/ using document root <none> and password file <none>
I20260806 03:02:48.569497 13467 fs_manager.cc:359] Metadata directory not provided
I20260806 03:02:48.569676 13467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:02:48.570068 13467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:02:48.573637 13467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785985105255475-32043-0/minicluster-data/master-0/data/instance:
uuid: "660bad56e7444e96bff441188d6421f1"
format_stamp: "Formatted at 2026-08-06 03:02:48 on dist-test-slave-dhph"
server_key: "7dfb0571b7779d23fe2b5d88d67004a1"
server_key_iv: "1880efa112cfc49563f63cc24249b48b"
server_key_version: "encryptionkey@0"
I20260806 03:02:48.574688 13467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785985105255475-32043-0/minicluster-data/master-0/wal/instance:
uuid: "660bad56e7444e96bff441188d6421f1"
format_stamp: "Formatted at 2026-08-06 03:02:48 on dist-test-slave-dhph"
server_key: "7dfb0571b7779d23fe2b5d88d67004a1"
server_key_iv: "1880efa112cfc49563f63cc24249b48b"
server_key_version: "encryptionkey@0"
I20260806 03:02:48.581164 13467 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260806 03:02:48.585536 13482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:02:48.587160 13467 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 03:02:48.587385 13467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785985105255475-32043-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785985105255475-32043-0/minicluster-data/master-0/wal
uuid: "660bad56e7444e96bff441188d6421f1"
format_stamp: "Formatted at 2026-08-06 03:02:48 on dist-test-slave-dhph"
server_key: "7dfb0571b7779d23fe2b5d88d67004a1"
server_key_iv: "1880efa112cfc49563f63cc24249b48b"
server_key_version: "encryptionkey@0"
I20260806 03:02:48.587613 13467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785985105255475-32043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785985105255475-32043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785985105255475-32043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:02:48.612263 13467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:02:48.617864 13467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:02:48.618232 13467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:02:48.642663 13467 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.254:38641
I20260806 03:02:48.642763 13535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.254:38641 every 8 connection(s)
I20260806 03:02:48.644156 13467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
I20260806 03:02:48.648200 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 13467
I20260806 03:02:48.648483 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785985105255475-32043-0/minicluster-data/master-0/wal/instance
I20260806 03:02:48.648780 13536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:02:48.649421 32043 external_mini_cluster.cc:1428] Setting key 57d12f5b9d5db709d40177a2fc5a2e8b
I20260806 03:02:48.661679 13536 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.74.254" port: 38641 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 03:02:48 dist-test-slave-dhph krb5kdc[13453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985368, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.254@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.74.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.253@KRBTEST.COM" created.
I20260806 03:02:48.689661 13539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:48.666328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:38641 (local address 127.0.0.1:51564)
0806 03:02:48.667350 (+ 1022us) negotiation.cc:107] Waiting for socket to connect
0806 03:02:48.667403 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 03:02:48.669870 (+ 2467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:02:48.670996 (+ 1126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:02:48.671003 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:02:48.671843 (+ 840us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:02:48.672214 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:02:48.672234 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:02:48.674012 (+ 1778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:02:48.674018 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:02:48.675345 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:02:48.675359 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:02:48.676491 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:02:48.676498 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:02:48.676716 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:02:48.677503 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:02:48.677743 (+ 240us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:02:48.680460 (+ 2717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:02:48.684362 (+ 3902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:02:48.684375 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:02:48.684389 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:02:48.684876 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:02:48.685306 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:02:48.685313 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:02:48.685317 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:02:48.685529 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:02:48.686997 (+ 1468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:02:48.687005 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:02:48.687259 (+ 254us) client_negotiation.cc:769] Sending connection context
0806 03:02:48.687564 (+ 305us) client_negotiation.cc:240] Negotiation successful
0806 03:02:48.688211 (+ 647us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":705,"mutex_wait_us":163,"thread_start_us":455,"threads_started":1}
I20260806 03:02:48.689684 13540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:48.668064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51564 (local address 127.31.74.254:38641)
0806 03:02:48.668922 (+ 858us) server_negotiation.cc:206] Beginning negotiation
0806 03:02:48.668932 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 03:02:48.669004 (+ 72us) server_negotiation.cc:407] Connection header received
0806 03:02:48.670362 (+ 1358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:02:48.670369 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:02:48.670737 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:02:48.670970 (+ 233us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:02:48.672444 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:48.673833 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:48.675502 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:48.676195 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:48.676442 (+ 247us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:02:48.680742 (+ 4300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:02:48.680750 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:02:48.680765 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:02:48.680799 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:02:48.684182 (+ 3383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:02:48.685032 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:02:48.685038 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:02:48.685041 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:02:48.685187 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:02:48.685656 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:02:48.685667 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:02:48.685670 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:02:48.686457 (+ 787us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:02:48.687078 (+ 621us) server_negotiation.cc:1035] Waiting for connection context
0806 03:02:48.687706 (+ 628us) server_negotiation.cc:299] Negotiation successful
0806 03:02:48.688078 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":398,"threads_started":1}
I20260806 03:02:48.693411 13536 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.74.253" port: 37437 } has no permanent_uuid. Determining permanent_uuid...
I20260806 03:02:48.694906 13539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:48.694083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.253:37437 (local address 127.0.0.1:58110)
0806 03:02:48.694401 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0806 03:02:48.694575 (+ 174us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.253:37437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260806 03:02:48.695331 13486 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.74.253:37437: connect: Connection refused (error 111)
I20260806 03:02:48.697659 13539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:48.697089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.253:37437 (local address 127.0.0.1:58124)
0806 03:02:48.697302 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0806 03:02:48.697431 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.253:37437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260806 03:02:48.697944 13536 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.74.253:37437: Network error: Client connection negotiation failed: client connection to 127.31.74.253:37437: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.253.keytab.
Entry for principal kudu/127.31.74.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.253.keytab.
I20260806 03:02:48.732956 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785985105255475-32043-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785985105255475-32043-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785985105255475-32043-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785985105255475-32043-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.253:37437
--webserver_interface=127.31.74.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.253.keytab
--principal=kudu/127.31.74.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:33253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.74.254:38641,127.31.74.253:37437,127.31.74.252:34107
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 03:02:48.747370 13536 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.74.253" port: 37437 } attempt: 1
I20260806 03:02:48.748764 13539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:48.748203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.253:37437 (local address 127.0.0.1:58130)
0806 03:02:48.748443 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0806 03:02:48.748596 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.253:37437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260806 03:02:48.750253 13539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:48.749600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.253:37437 (local address 127.0.0.1:58144)
0806 03:02:48.749888 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0806 03:02:48.750025 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.253:37437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
W20260806 03:02:48.750687 13536 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.74.253:37437: Network error: Client connection negotiation failed: client connection to 127.31.74.253:37437: connect: Connection refused (error 111)
I20260806 03:02:48.819065 13536 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.74.253" port: 37437 } attempt: 2
I20260806 03:02:48.820649 13539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:48.819958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.253:37437 (local address 127.0.0.1:58158)
0806 03:02:48.820244 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0806 03:02:48.820339 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.253:37437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260806 03:02:48.822515 13539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:48.821866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.253:37437 (local address 127.0.0.1:58172)
0806 03:02:48.822249 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0806 03:02:48.822361 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.253:37437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260806 03:02:48.822865 13536 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.74.253:37437: Network error: Client connection negotiation failed: client connection to 127.31.74.253:37437: connect: Connection refused (error 111)
I20260806 03:02:48.914311 13536 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.74.253" port: 37437 } attempt: 3
I20260806 03:02:48.915721 13539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:48.915206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.253:37437 (local address 127.0.0.1:58182)
0806 03:02:48.915474 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0806 03:02:48.915589 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.253:37437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
I20260806 03:02:48.916993 13539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:48.916548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.253:37437 (local address 127.0.0.1:58188)
0806 03:02:48.916731 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0806 03:02:48.916815 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.253:37437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260806 03:02:48.917347 13536 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.74.253:37437: Network error: Client connection negotiation failed: client connection to 127.31.74.253:37437: connect: Connection refused (error 111)
I20260806 03:02:49.060755 13536 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.74.253" port: 37437 } attempt: 4
I20260806 03:02:49.062510 13539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:49.061782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.253:37437 (local address 127.0.0.1:58204)
0806 03:02:49.062225 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0806 03:02:49.062334 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.253:37437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":274}
I20260806 03:02:49.063838 13539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:49.063371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.253:37437 (local address 127.0.0.1:58208)
0806 03:02:49.063538 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0806 03:02:49.063649 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.253:37437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260806 03:02:49.064114 13536 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.74.253:37437: Network error: Client connection negotiation failed: client connection to 127.31.74.253:37437: connect: Connection refused (error 111)
W20260806 03:02:49.139420 13543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:02:49.139789 13543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:02:49.139956 13543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:02:49.148532 13543 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:02:49.148653 13543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:02:49.148725 13543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:02:49.148818 13543 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:02:49.148860 13543 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:02:49.160894 13543 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:33253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785985105255475-32043-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785985105255475-32043-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.31.74.254:38641,127.31.74.253:37437,127.31.74.252:34107
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.253.keytab
--principal=kudu/127.31.74.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.253:37437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785985105255475-32043-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.31.74.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785985105255475-32043-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:02:49.162426 13543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:02:49.163851 13543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:02:49.179888 13548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:02:49.179838 13550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:02:49.179811 13552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:02:49.180398 13543 server_base.cc:1034] running on GCE node
I20260806 03:02:49.181802 13543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:02:49.183491 13543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:02:49.184916 13543 hybrid_clock.cc:648] HybridClock initialized: now 1785985369184734 us; error 56 us; skew 500 ppm
I20260806 03:02:49.185388 13543 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 03:02:49 dist-test-slave-dhph krb5kdc[13453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985369, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:02:49.194276 13543 init.cc:377] Logged in from keytab as kudu/127.31.74.253@KRBTEST.COM (short username kudu)
I20260806 03:02:49.196313 13543 webserver.cc:460] Webserver started at http://127.31.74.253:35517/ using document root <none> and password file <none>
I20260806 03:02:49.197105 13543 fs_manager.cc:359] Metadata directory not provided
I20260806 03:02:49.197233 13543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:02:49.197539 13543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:02:49.201547 13543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785985105255475-32043-0/minicluster-data/master-1/data/instance:
uuid: "c29e78f76c6f4116998c665614de9e90"
format_stamp: "Formatted at 2026-08-06 03:02:49 on dist-test-slave-dhph"
server_key: "a0ec52b1d0d8c4d9acf29233620bb9eb"
server_key_iv: "0ed4ab8347b57bb73d1e3c2e07d2c5d8"
server_key_version: "encryptionkey@0"
I20260806 03:02:49.202544 13543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785985105255475-32043-0/minicluster-data/master-1/wal/instance:
uuid: "c29e78f76c6f4116998c665614de9e90"
format_stamp: "Formatted at 2026-08-06 03:02:49 on dist-test-slave-dhph"
server_key: "a0ec52b1d0d8c4d9acf29233620bb9eb"
server_key_iv: "0ed4ab8347b57bb73d1e3c2e07d2c5d8"
server_key_version: "encryptionkey@0"
I20260806 03:02:49.208948 13543 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260806 03:02:49.212850 13559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:02:49.215024 13543 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 03:02:49.215269 13543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785985105255475-32043-0/minicluster-data/master-1/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785985105255475-32043-0/minicluster-data/master-1/wal
uuid: "c29e78f76c6f4116998c665614de9e90"
format_stamp: "Formatted at 2026-08-06 03:02:49 on dist-test-slave-dhph"
server_key: "a0ec52b1d0d8c4d9acf29233620bb9eb"
server_key_iv: "0ed4ab8347b57bb73d1e3c2e07d2c5d8"
server_key_version: "encryptionkey@0"
I20260806 03:02:49.215497 13543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785985105255475-32043-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785985105255475-32043-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785985105255475-32043-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:02:49.239982 13543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:02:49.245576 13543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:02:49.245952 13543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:02:49.270022 13543 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.253:37437
I20260806 03:02:49.270041 13611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.253:37437 every 8 connection(s)
I20260806 03:02:49.271793 13543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785985105255475-32043-0/minicluster-data/master-1/data/info.pb
I20260806 03:02:49.276636 13612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:02:49.281606 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 13543
I20260806 03:02:49.281842 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785985105255475-32043-0/minicluster-data/master-1/wal/instance
I20260806 03:02:49.282454 32043 external_mini_cluster.cc:1428] Setting key 8ac6789bfaf2eef386d8b819482193c1
I20260806 03:02:49.286324 13612 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.74.254" port: 38641 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 03:02:49 dist-test-slave-dhph krb5kdc[13453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985369, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.253@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.74.252@KRBTEST.COM; defaulting to no policy
I20260806 03:02:49.318501 13614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:49.290446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51568 (local address 127.31.74.254:38641)
0806 03:02:49.291185 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0806 03:02:49.291194 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 03:02:49.293006 (+ 1812us) server_negotiation.cc:407] Connection header received
0806 03:02:49.296469 (+ 3463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:02:49.296477 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:02:49.296590 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:02:49.296738 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:02:49.299593 (+ 2855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:49.300888 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:49.303176 (+ 2288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:49.304233 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:49.304399 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:02:49.308860 (+ 4461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:02:49.308870 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:02:49.308878 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:02:49.308935 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:02:49.313762 (+ 4827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:02:49.315272 (+ 1510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:02:49.315282 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:02:49.315287 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:02:49.315478 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:02:49.316203 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:02:49.316207 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:02:49.316210 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:02:49.316582 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:02:49.316734 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0806 03:02:49.317835 (+ 1101us) server_negotiation.cc:299] Negotiation successful
0806 03:02:49.318050 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":390,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.252@KRBTEST.COM" created.
I20260806 03:02:49.319554 13615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:49.291267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:38641 (local address 127.0.0.1:51568)
0806 03:02:49.292706 (+ 1439us) negotiation.cc:107] Waiting for socket to connect
0806 03:02:49.292782 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0806 03:02:49.296039 (+ 3257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:02:49.297065 (+ 1026us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:02:49.297075 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:02:49.297877 (+ 802us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:02:49.299221 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:02:49.299258 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:02:49.301185 (+ 1927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:02:49.301191 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:02:49.302957 (+ 1766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:02:49.302976 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:02:49.304584 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:02:49.304595 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:02:49.304999 (+ 404us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:02:49.305789 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:02:49.305821 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:02:49.308542 (+ 2721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:02:49.314148 (+ 5606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:02:49.314158 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:02:49.314178 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:02:49.315064 (+ 886us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:02:49.315645 (+ 581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:02:49.315649 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:02:49.315652 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:02:49.316028 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:02:49.316809 (+ 781us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:02:49.316827 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:02:49.317418 (+ 591us) client_negotiation.cc:769] Sending connection context
0806 03:02:49.317675 (+ 257us) client_negotiation.cc:240] Negotiation successful
0806 03:02:49.318091 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1012,"thread_start_us":542,"threads_started":1}
I20260806 03:02:49.322618 13612 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.74.253" port: 37437 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 03:02:49 dist-test-slave-dhph krb5kdc[13453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985369, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.253@KRBTEST.COM for kudu/127.31.74.253@KRBTEST.COM
I20260806 03:02:49.340497 13617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:49.323833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58214 (local address 127.31.74.253:37437)
0806 03:02:49.324584 (+ 751us) server_negotiation.cc:206] Beginning negotiation
0806 03:02:49.324591 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 03:02:49.324618 (+ 27us) server_negotiation.cc:407] Connection header received
0806 03:02:49.324668 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:02:49.324673 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:02:49.324871 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:02:49.325126 (+ 255us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:02:49.325933 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:49.327159 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:49.328372 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:49.329050 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:49.329225 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:02:49.332825 (+ 3600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:02:49.332844 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:02:49.332899 (+ 55us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:02:49.332951 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:02:49.336907 (+ 3956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:02:49.337972 (+ 1065us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:02:49.337983 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:02:49.337985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:02:49.338124 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:02:49.338696 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:02:49.338700 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:02:49.338702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:02:49.339135 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:02:49.339324 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0806 03:02:49.339799 (+ 475us) server_negotiation.cc:299] Negotiation successful
0806 03:02:49.340169 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":121,"server-negotiator.queue_time_us":598,"thread_start_us":465,"threads_started":1}
I20260806 03:02:49.340546 13615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:49.323561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.253:37437 (local address 127.0.0.1:58214)
0806 03:02:49.323770 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0806 03:02:49.323787 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 03:02:49.323894 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:02:49.325123 (+ 1229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:02:49.325128 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:02:49.325620 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:02:49.325756 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:02:49.325766 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:02:49.327306 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:02:49.327313 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:02:49.328128 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:02:49.328139 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:02:49.329247 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:02:49.329254 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:02:49.329494 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:02:49.330205 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:02:49.330224 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:02:49.332488 (+ 2264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:02:49.337126 (+ 4638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:02:49.337135 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:02:49.337141 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:02:49.337786 (+ 645us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:02:49.338276 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:02:49.338284 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:02:49.338290 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:02:49.338493 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:02:49.339286 (+ 793us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:02:49.339294 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:02:49.339472 (+ 178us) client_negotiation.cc:769] Sending connection context
0806 03:02:49.339640 (+ 168us) client_negotiation.cc:240] Negotiation successful
0806 03:02:49.340010 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260806 03:02:49.343695 13612 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.74.252" port: 34107 } has no permanent_uuid. Determining permanent_uuid...
I20260806 03:02:49.345155 13615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:49.344528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.252:34107 (local address 127.0.0.1:46920)
0806 03:02:49.344869 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0806 03:02:49.344991 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.252:34107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193}
W20260806 03:02:49.345429 13563 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.74.252:34107: connect: Connection refused (error 111)
I20260806 03:02:49.348373 13615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:49.347800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.252:34107 (local address 127.0.0.1:46936)
0806 03:02:49.348056 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0806 03:02:49.348186 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.252:34107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260806 03:02:49.348817 13612 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.74.252:34107: Network error: Client connection negotiation failed: client connection to 127.31.74.252:34107: connect: Connection refused (error 111)
I20260806 03:02:49.363492 13536 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.74.253" port: 37437 } attempt: 5
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.252.keytab.
Entry for principal kudu/127.31.74.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.252.keytab.
I20260806 03:02:49.369445 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785985105255475-32043-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785985105255475-32043-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785985105255475-32043-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785985105255475-32043-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.252:34107
--webserver_interface=127.31.74.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.252.keytab
--principal=kudu/127.31.74.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:33253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.74.254:38641,127.31.74.253:37437,127.31.74.252:34107
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 03:02:49 dist-test-slave-dhph krb5kdc[13453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985368, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.254@KRBTEST.COM for kudu/127.31.74.253@KRBTEST.COM
I20260806 03:02:49.383257 13539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:49.364481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.253:37437 (local address 127.0.0.1:58222)
0806 03:02:49.364742 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0806 03:02:49.364762 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 03:02:49.364938 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:02:49.365379 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:02:49.365384 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:02:49.365821 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:02:49.366058 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:02:49.366069 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:02:49.367921 (+ 1852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:02:49.367932 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:02:49.369593 (+ 1661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:02:49.369615 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:02:49.370681 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:02:49.370689 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:02:49.370974 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:02:49.371999 (+ 1025us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:02:49.372030 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:02:49.375008 (+ 2978us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:02:49.380362 (+ 5354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:02:49.380372 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:02:49.380375 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:02:49.380933 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:02:49.381404 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:02:49.381410 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:02:49.381414 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:02:49.381570 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:02:49.382156 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:02:49.382161 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:02:49.382332 (+ 171us) client_negotiation.cc:769] Sending connection context
0806 03:02:49.382489 (+ 157us) client_negotiation.cc:240] Negotiation successful
0806 03:02:49.382890 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122}
I20260806 03:02:49.383251 13617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:49.364754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58222 (local address 127.31.74.253:37437)
0806 03:02:49.364996 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0806 03:02:49.365005 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 03:02:49.365027 (+ 22us) server_negotiation.cc:407] Connection header received
0806 03:02:49.365121 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:02:49.365128 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:02:49.365230 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:02:49.365343 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:02:49.366243 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:49.367621 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:49.369799 (+ 2178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:49.370420 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:49.370560 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:02:49.375493 (+ 4933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:02:49.375500 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:02:49.375503 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:02:49.375546 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:02:49.380066 (+ 4520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:02:49.381111 (+ 1045us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:02:49.381122 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:02:49.381124 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:02:49.381269 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:02:49.381693 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:02:49.381699 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:02:49.381702 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:02:49.382028 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:02:49.382197 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0806 03:02:49.382628 (+ 431us) server_negotiation.cc:299] Negotiation successful
0806 03:02:49.382890 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260806 03:02:49.384717 13536 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.74.252" port: 34107 } has no permanent_uuid. Determining permanent_uuid...
I20260806 03:02:49.386085 13539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:49.385495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.252:34107 (local address 127.0.0.1:46940)
0806 03:02:49.385724 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0806 03:02:49.385857 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.252:34107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260806 03:02:49.388155 13539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:49.387211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.252:34107 (local address 127.0.0.1:46948)
0806 03:02:49.387629 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0806 03:02:49.387842 (+ 213us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.252:34107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260806 03:02:49.388633 13536 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.74.252:34107: Network error: Client connection negotiation failed: client connection to 127.31.74.252:34107: connect: Connection refused (error 111)
I20260806 03:02:49.398625 13612 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.74.252" port: 34107 } attempt: 1
I20260806 03:02:49.400401 13615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:49.399826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.252:34107 (local address 127.0.0.1:46956)
0806 03:02:49.400101 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0806 03:02:49.400226 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.252:34107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139}
I20260806 03:02:49.402737 13615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:49.401721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.252:34107 (local address 127.0.0.1:46970)
0806 03:02:49.402388 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0806 03:02:49.402492 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.252:34107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400}
W20260806 03:02:49.402992 13612 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.74.252:34107: Network error: Client connection negotiation failed: client connection to 127.31.74.252:34107: connect: Connection refused (error 111)
I20260806 03:02:49.440030 13536 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.74.252" port: 34107 } attempt: 1
I20260806 03:02:49.441640 13539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:49.440922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.252:34107 (local address 127.0.0.1:46982)
0806 03:02:49.441220 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0806 03:02:49.441380 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.252:34107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260806 03:02:49.442962 13539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:49.442519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.252:34107 (local address 127.0.0.1:46998)
0806 03:02:49.442716 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0806 03:02:49.442820 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.252:34107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260806 03:02:49.443248 13536 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.74.252:34107: Network error: Client connection negotiation failed: client connection to 127.31.74.252:34107: connect: Connection refused (error 111)
I20260806 03:02:49.471276 13612 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.74.252" port: 34107 } attempt: 2
I20260806 03:02:49.472805 13615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:49.472063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.252:34107 (local address 127.0.0.1:47002)
0806 03:02:49.472295 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0806 03:02:49.472385 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.252:34107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260806 03:02:49.473883 13615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:49.473529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.252:34107 (local address 127.0.0.1:47008)
0806 03:02:49.473677 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0806 03:02:49.473763 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.252:34107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260806 03:02:49.474167 13612 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.74.252:34107: Network error: Client connection negotiation failed: client connection to 127.31.74.252:34107: connect: Connection refused (error 111)
I20260806 03:02:49.511596 13536 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.74.252" port: 34107 } attempt: 2
I20260806 03:02:49.513556 13539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:49.512728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.252:34107 (local address 127.0.0.1:47010)
0806 03:02:49.513155 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0806 03:02:49.513259 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.252:34107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143}
I20260806 03:02:49.515321 13539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:49.514893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.252:34107 (local address 127.0.0.1:47012)
0806 03:02:49.515080 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0806 03:02:49.515189 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.252:34107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260806 03:02:49.515559 13536 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.74.252:34107: Network error: Client connection negotiation failed: client connection to 127.31.74.252:34107: connect: Connection refused (error 111)
I20260806 03:02:49.565472 13612 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.74.252" port: 34107 } attempt: 3
I20260806 03:02:49.567199 13615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:49.566617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.252:34107 (local address 127.0.0.1:47014)
0806 03:02:49.566908 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0806 03:02:49.567004 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.252:34107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260806 03:02:49.568471 13615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:49.567975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.252:34107 (local address 127.0.0.1:47016)
0806 03:02:49.568224 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0806 03:02:49.568334 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.252:34107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
W20260806 03:02:49.568715 13612 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.74.252:34107: Network error: Client connection negotiation failed: client connection to 127.31.74.252:34107: connect: Connection refused (error 111)
I20260806 03:02:49.621794 13536 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.74.252" port: 34107 } attempt: 3
I20260806 03:02:49.623000 13539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:49.622499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.252:34107 (local address 127.0.0.1:47018)
0806 03:02:49.622735 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0806 03:02:49.622831 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.252:34107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260806 03:02:49.624240 13539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:49.623780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.252:34107 (local address 127.0.0.1:47034)
0806 03:02:49.624048 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0806 03:02:49.624125 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.252:34107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161}
W20260806 03:02:49.624454 13536 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.74.252:34107: Network error: Client connection negotiation failed: client connection to 127.31.74.252:34107: connect: Connection refused (error 111)
I20260806 03:02:49.711989 13612 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.74.252" port: 34107 } attempt: 4
I20260806 03:02:49.713687 13615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:49.712801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.252:34107 (local address 127.0.0.1:47040)
0806 03:02:49.713010 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0806 03:02:49.713311 (+ 301us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.252:34107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260806 03:02:49.715159 13615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:49.714698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.252:34107 (local address 127.0.0.1:47050)
0806 03:02:49.714876 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0806 03:02:49.714970 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.252:34107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260806 03:02:49.715444 13612 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.74.252:34107: Network error: Client connection negotiation failed: client connection to 127.31.74.252:34107: connect: Connection refused (error 111)
W20260806 03:02:49.776564 13621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:02:49.776891 13621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:02:49.777063 13621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:02:49.784775 13621 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:02:49.784917 13621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:02:49.785005 13621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:02:49.785096 13621 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:02:49.785166 13621 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:02:49.796655 13621 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:33253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785985105255475-32043-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785985105255475-32043-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.31.74.254:38641,127.31.74.253:37437,127.31.74.252:34107
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.252.keytab
--principal=kudu/127.31.74.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.252:34107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785985105255475-32043-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.31.74.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785985105255475-32043-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:02:49.798296 13621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:02:49.799726 13621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 03:02:49.801687 13536 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.74.252" port: 34107 } attempt: 4
I20260806 03:02:49.802960 13539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:49.802480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.252:34107 (local address 127.0.0.1:47060)
0806 03:02:49.802710 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0806 03:02:49.802808 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.252:34107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260806 03:02:49.804072 13539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:49.803696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.252:34107 (local address 127.0.0.1:47070)
0806 03:02:49.803866 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0806 03:02:49.803942 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.252:34107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260806 03:02:49.804358 13536 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.74.252:34107: Network error: Client connection negotiation failed: client connection to 127.31.74.252:34107: connect: Connection refused (error 111)
W20260806 03:02:49.814062 13626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:02:49.814291 13627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:02:49.814404 13629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:02:49.815894 13621 server_base.cc:1034] running on GCE node
I20260806 03:02:49.816810 13621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:02:49.818506 13621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:02:49.819802 13621 hybrid_clock.cc:648] HybridClock initialized: now 1785985369819749 us; error 53 us; skew 500 ppm
I20260806 03:02:49.820246 13621 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 03:02:49 dist-test-slave-dhph krb5kdc[13453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985369, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:02:49.824230 13621 init.cc:377] Logged in from keytab as kudu/127.31.74.252@KRBTEST.COM (short username kudu)
I20260806 03:02:49.826221 13621 webserver.cc:460] Webserver started at http://127.31.74.252:40155/ using document root <none> and password file <none>
I20260806 03:02:49.827206 13621 fs_manager.cc:359] Metadata directory not provided
I20260806 03:02:49.827391 13621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:02:49.827790 13621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:02:49.832190 13621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785985105255475-32043-0/minicluster-data/master-2/data/instance:
uuid: "3af78c2f583343749f10775295433664"
format_stamp: "Formatted at 2026-08-06 03:02:49 on dist-test-slave-dhph"
server_key: "4fff57aa7f6cc42cd202b73ade2cb812"
server_key_iv: "581b56ba40cd0d2f10659cbac6e86de9"
server_key_version: "encryptionkey@0"
I20260806 03:02:49.833108 13621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785985105255475-32043-0/minicluster-data/master-2/wal/instance:
uuid: "3af78c2f583343749f10775295433664"
format_stamp: "Formatted at 2026-08-06 03:02:49 on dist-test-slave-dhph"
server_key: "4fff57aa7f6cc42cd202b73ade2cb812"
server_key_iv: "581b56ba40cd0d2f10659cbac6e86de9"
server_key_version: "encryptionkey@0"
I20260806 03:02:49.839442 13621 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260806 03:02:49.845084 13637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:02:49.847184 13621 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 03:02:49.847437 13621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785985105255475-32043-0/minicluster-data/master-2/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785985105255475-32043-0/minicluster-data/master-2/wal
uuid: "3af78c2f583343749f10775295433664"
format_stamp: "Formatted at 2026-08-06 03:02:49 on dist-test-slave-dhph"
server_key: "4fff57aa7f6cc42cd202b73ade2cb812"
server_key_iv: "581b56ba40cd0d2f10659cbac6e86de9"
server_key_version: "encryptionkey@0"
I20260806 03:02:49.847699 13621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785985105255475-32043-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785985105255475-32043-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785985105255475-32043-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:02:49.873389 13621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:02:49.879583 13621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:02:49.879942 13621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:02:49.905453 13690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.252:34107 every 8 connection(s)
I20260806 03:02:49.905454 13621 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.252:34107
I20260806 03:02:49.907388 13621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785985105255475-32043-0/minicluster-data/master-2/data/info.pb
I20260806 03:02:49.908244 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 13621
I20260806 03:02:49.908457 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785985105255475-32043-0/minicluster-data/master-2/wal/instance
I20260806 03:02:49.909035 32043 external_mini_cluster.cc:1428] Setting key 65d57d805546ee06f8289d10f4069238
I20260806 03:02:49.912675 13691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:02:49.925278 13691 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.74.254" port: 38641 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 03:02:49 dist-test-slave-dhph krb5kdc[13453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985367, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.74.252@KRBTEST.COM
Aug 06 03:02:49 dist-test-slave-dhph krb5kdc[13453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985369, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.252@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 03:02:49.943096 13694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:49.911375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47080 (local address 127.31.74.252:34107)
0806 03:02:49.912470 (+ 1095us) server_negotiation.cc:206] Beginning negotiation
0806 03:02:49.912488 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 03:02:49.912544 (+ 56us) server_negotiation.cc:407] Connection header received
0806 03:02:49.915084 (+ 2540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:02:49.915115 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:02:49.915661 (+ 546us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:02:49.916175 (+ 514us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:02:49.918255 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:49.921502 (+ 3247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:49.923577 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:49.924716 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:49.924923 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:02:49.932527 (+ 7604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:02:49.932537 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:02:49.932549 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:02:49.932593 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:02:49.936044 (+ 3451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:02:49.937048 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:02:49.937057 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:02:49.937068 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:02:49.937278 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:02:49.938077 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:02:49.938083 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:02:49.938086 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:02:49.939038 (+ 952us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:02:49.939473 (+ 435us) server_negotiation.cc:1035] Waiting for connection context
0806 03:02:49.940069 (+ 596us) server_negotiation.cc:299] Negotiation successful
0806 03:02:49.940672 (+ 603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":739,"thread_start_us":384,"threads_started":1}
I20260806 03:02:49.952149 13696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:49.929253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:38641 (local address 127.0.0.1:51570)
0806 03:02:49.930375 (+ 1122us) negotiation.cc:107] Waiting for socket to connect
0806 03:02:49.930428 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 03:02:49.930663 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:02:49.931314 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:02:49.931322 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:02:49.932846 (+ 1524us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:02:49.933217 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:02:49.933232 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:02:49.934524 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:02:49.934530 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:02:49.935908 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:02:49.935922 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:02:49.936931 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:02:49.936938 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:02:49.937143 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:02:49.938099 (+ 956us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:02:49.938123 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:02:49.941013 (+ 2890us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:02:49.947016 (+ 6003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:02:49.947028 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:02:49.947035 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:02:49.948463 (+ 1428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:02:49.949620 (+ 1157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:02:49.949630 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:02:49.949636 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:02:49.949910 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:02:49.950926 (+ 1016us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:02:49.950933 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:02:49.951125 (+ 192us) client_negotiation.cc:769] Sending connection context
0806 03:02:49.951307 (+ 182us) client_negotiation.cc:240] Negotiation successful
0806 03:02:49.951685 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":815,"thread_start_us":499,"threads_started":1}
I20260806 03:02:49.952252 13695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:49.928945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51570 (local address 127.31.74.254:38641)
0806 03:02:49.929705 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0806 03:02:49.929727 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0806 03:02:49.930659 (+ 932us) server_negotiation.cc:407] Connection header received
0806 03:02:49.930809 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:02:49.930814 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:02:49.931001 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:02:49.931200 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:02:49.933435 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:49.934322 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:49.936094 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:49.936681 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:49.936889 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:02:49.941260 (+ 4371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:02:49.941268 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:02:49.941272 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:02:49.942157 (+ 885us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:02:49.946787 (+ 4630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:02:49.949012 (+ 2225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:02:49.949030 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:02:49.949038 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:02:49.949398 (+ 360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:02:49.950132 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:02:49.950139 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:02:49.950142 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:02:49.950677 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:02:49.950902 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0806 03:02:49.951487 (+ 585us) server_negotiation.cc:299] Negotiation successful
0806 03:02:49.951737 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":351,"threads_started":1}
I20260806 03:02:49.954178 13691 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.74.253" port: 37437 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 03:02:49 dist-test-slave-dhph krb5kdc[13453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985367, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
Aug 06 03:02:49 dist-test-slave-dhph krb5kdc[13453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985369, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.252@KRBTEST.COM for kudu/127.31.74.253@KRBTEST.COM
I20260806 03:02:49.970737 13697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:49.948027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51572 (local address 127.31.74.254:38641)
0806 03:02:49.949674 (+ 1647us) server_negotiation.cc:206] Beginning negotiation
0806 03:02:49.949680 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 03:02:49.949710 (+ 30us) server_negotiation.cc:407] Connection header received
0806 03:02:49.949793 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:02:49.949799 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:02:49.949888 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:02:49.950034 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:02:49.951907 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:49.953305 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:49.955940 (+ 2635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:49.956834 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:49.956995 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:02:49.963285 (+ 6290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:02:49.963292 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:02:49.963295 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:02:49.963346 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:02:49.966789 (+ 3443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:02:49.967690 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:02:49.967695 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:02:49.967698 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:02:49.967852 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:02:49.968316 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:02:49.968323 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:02:49.968326 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:02:49.968886 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:02:49.969508 (+ 622us) server_negotiation.cc:1035] Waiting for connection context
0806 03:02:49.969901 (+ 393us) server_negotiation.cc:299] Negotiation successful
0806 03:02:49.970175 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1409,"thread_start_us":1163,"threads_started":1}
I20260806 03:02:49.977871 13698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:49.955484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58236 (local address 127.31.74.253:37437)
0806 03:02:49.956206 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0806 03:02:49.956215 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 03:02:49.956391 (+ 176us) server_negotiation.cc:407] Connection header received
0806 03:02:49.956477 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:02:49.956482 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:02:49.956574 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:02:49.956710 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:02:49.959653 (+ 2943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:49.960892 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:49.963413 (+ 2521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:49.963904 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:49.964913 (+ 1009us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:02:49.969452 (+ 4539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:02:49.969458 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:02:49.969461 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:02:49.969498 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:02:49.973071 (+ 3573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:02:49.974117 (+ 1046us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:02:49.974126 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:02:49.974133 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:02:49.974395 (+ 262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:02:49.975611 (+ 1216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:02:49.975617 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:02:49.975620 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:02:49.976212 (+ 592us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:02:49.976405 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0806 03:02:49.977291 (+ 886us) server_negotiation.cc:299] Negotiation successful
0806 03:02:49.977503 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":517,"threads_started":1}
I20260806 03:02:49.977901 13696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:49.955326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.253:37437 (local address 127.0.0.1:58236)
0806 03:02:49.955535 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0806 03:02:49.955558 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 03:02:49.955688 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:02:49.957106 (+ 1418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:02:49.957121 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:02:49.958804 (+ 1683us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:02:49.959417 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:02:49.959430 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:02:49.961100 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:02:49.961104 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:02:49.963108 (+ 2004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:02:49.963139 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:02:49.964934 (+ 1795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:02:49.964943 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:02:49.965439 (+ 496us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:02:49.966296 (+ 857us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:02:49.966324 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:02:49.969187 (+ 2863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:02:49.973281 (+ 4094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:02:49.973290 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:02:49.973296 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:02:49.973932 (+ 636us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:02:49.974544 (+ 612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:02:49.974549 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:02:49.974553 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:02:49.974797 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:02:49.976513 (+ 1716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:02:49.976520 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:02:49.976963 (+ 443us) client_negotiation.cc:769] Sending connection context
0806 03:02:49.977125 (+ 162us) client_negotiation.cc:240] Negotiation successful
0806 03:02:49.977527 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260806 03:02:49.979521 13691 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.74.252" port: 34107 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 03:02:49 dist-test-slave-dhph krb5kdc[13453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985367, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.74.253@KRBTEST.COM
Aug 06 03:02:49 dist-test-slave-dhph krb5kdc[13453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985369, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.252@KRBTEST.COM for kudu/127.31.74.252@KRBTEST.COM
I20260806 03:02:49.993888 13699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:49.973736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58246 (local address 127.31.74.253:37437)
0806 03:02:49.974647 (+ 911us) server_negotiation.cc:206] Beginning negotiation
0806 03:02:49.974653 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 03:02:49.974680 (+ 27us) server_negotiation.cc:407] Connection header received
0806 03:02:49.974732 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:02:49.974736 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:02:49.974812 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:02:49.974922 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:02:49.977368 (+ 2446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:49.978749 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:49.980180 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:49.981059 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:49.981200 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:02:49.986125 (+ 4925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:02:49.986131 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:02:49.986134 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:02:49.986171 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:02:49.989542 (+ 3371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:02:49.990617 (+ 1075us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:02:49.990634 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:02:49.990639 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:02:49.990839 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:02:49.991424 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:02:49.991430 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:02:49.991435 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:02:49.992105 (+ 670us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:02:49.992427 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0806 03:02:49.992717 (+ 290us) server_negotiation.cc:299] Negotiation successful
0806 03:02:49.993339 (+ 622us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":758,"thread_start_us":482,"threads_started":1}
I20260806 03:02:49.998885 13696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:49.980763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.252:34107 (local address 127.0.0.1:47092)
0806 03:02:49.980927 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0806 03:02:49.980955 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0806 03:02:49.981083 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:02:49.981358 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:02:49.981362 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:02:49.981958 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:02:49.982264 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:02:49.982286 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:02:49.983402 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:02:49.983410 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:02:49.984301 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:02:49.984319 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:02:49.985037 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:02:49.985042 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:02:49.985223 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:02:49.986362 (+ 1139us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:02:49.986383 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:02:49.989031 (+ 2648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:02:49.995397 (+ 6366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:02:49.995404 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:02:49.995409 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:02:49.996175 (+ 766us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:02:49.996808 (+ 633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:02:49.996814 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:02:49.996817 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:02:49.997099 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:02:49.997669 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:02:49.997675 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:02:49.997857 (+ 182us) client_negotiation.cc:769] Sending connection context
0806 03:02:49.998064 (+ 207us) client_negotiation.cc:240] Negotiation successful
0806 03:02:49.998424 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260806 03:02:49.999076 13694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:49.980932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47092 (local address 127.31.74.252:34107)
0806 03:02:49.981092 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0806 03:02:49.981097 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 03:02:49.981115 (+ 18us) server_negotiation.cc:407] Connection header received
0806 03:02:49.981171 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:02:49.981176 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:02:49.981236 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:02:49.981332 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:02:49.982425 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:49.983278 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:49.984457 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:49.984864 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:49.985014 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:02:49.989260 (+ 4246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:02:49.989269 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:02:49.989275 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:02:49.989320 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:02:49.995154 (+ 5834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:02:49.996309 (+ 1155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:02:49.996315 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:02:49.996319 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:02:49.996479 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:02:49.997215 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:02:49.997219 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:02:49.997221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:02:49.997549 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:02:49.997671 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0806 03:02:49.998185 (+ 514us) server_negotiation.cc:299] Negotiation successful
0806 03:02:49.998644 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":167,"server-negotiator.queue_time_us":62}
I20260806 03:02:50.002822 32043 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 03:02:50.007640 13691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3af78c2f583343749f10775295433664: Bootstrap starting.
I20260806 03:02:50.011690 13691 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3af78c2f583343749f10775295433664: Neither blocks nor log segments found. Creating new log.
I20260806 03:02:50.013682 13691 log.cc:824] T 00000000000000000000000000000000 P 3af78c2f583343749f10775295433664: Log is configured to *not* fsync() on all Append() calls
I20260806 03:02:50.014652 13612 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.74.252" port: 34107 } attempt: 5
I20260806 03:02:50.047377 13697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:50.011325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51588 (local address 127.31.74.254:38641)
0806 03:02:50.011591 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0806 03:02:50.011598 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 03:02:50.012261 (+ 663us) server_negotiation.cc:407] Connection header received
0806 03:02:50.012583 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:02:50.012589 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:02:50.012689 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:02:50.012771 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:02:50.016686 (+ 3915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:50.018135 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:50.023335 (+ 5200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:50.024334 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:50.024480 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:02:50.034143 (+ 9663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:02:50.034158 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:02:50.034167 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:02:50.034260 (+ 93us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:02:50.040300 (+ 6040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:02:50.042063 (+ 1763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:02:50.042072 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:02:50.042079 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:02:50.042302 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:02:50.044788 (+ 2486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:02:50.044795 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:02:50.044800 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:02:50.045280 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:02:50.045570 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0806 03:02:50.046362 (+ 792us) server_negotiation.cc:299] Negotiation successful
0806 03:02:50.046924 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
Aug 06 03:02:50 dist-test-slave-dhph krb5kdc[13453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985369, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.253@KRBTEST.COM for kudu/127.31.74.252@KRBTEST.COM
I20260806 03:02:50.049985 13691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3af78c2f583343749f10775295433664: No bootstrap required, opened a new log
I20260806 03:02:50.053732 13694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:50.013507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47100 (local address 127.31.74.252:34107)
0806 03:02:50.013738 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0806 03:02:50.013743 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 03:02:50.024927 (+ 11184us) server_negotiation.cc:407] Connection header received
0806 03:02:50.026150 (+ 1223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:02:50.026161 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:02:50.026284 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:02:50.026379 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:02:50.030568 (+ 4189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:50.032071 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:50.034910 (+ 2839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:50.036494 (+ 1584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:50.036761 (+ 267us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:02:50.042127 (+ 5366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:02:50.042134 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:02:50.042139 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:02:50.042187 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:02:50.048753 (+ 6566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:02:50.050019 (+ 1266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:02:50.050026 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:02:50.050030 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:02:50.050230 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:02:50.050840 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:02:50.050846 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:02:50.050849 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:02:50.051388 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:02:50.051683 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0806 03:02:50.052492 (+ 809us) server_negotiation.cc:299] Negotiation successful
0806 03:02:50.053049 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 03:02:50.057655 13699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:50.012399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58248 (local address 127.31.74.253:37437)
0806 03:02:50.012582 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0806 03:02:50.012588 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 03:02:50.013480 (+ 892us) server_negotiation.cc:407] Connection header received
0806 03:02:50.013761 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:02:50.013767 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:02:50.013868 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:02:50.013945 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:02:50.027706 (+ 13761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:50.028816 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:50.034351 (+ 5535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:50.035152 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:50.035274 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:02:50.047418 (+ 12144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:02:50.047428 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:02:50.047435 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:02:50.047491 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:02:50.051866 (+ 4375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:02:50.053078 (+ 1212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:02:50.053086 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:02:50.053092 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:02:50.053265 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:02:50.054064 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:02:50.054073 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:02:50.054079 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:02:50.054622 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:02:50.054943 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0806 03:02:50.057040 (+ 2097us) server_negotiation.cc:299] Negotiation successful
0806 03:02:50.057307 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 03:02:50.060992 13708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:50.016344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47116 (local address 127.31.74.252:34107)
0806 03:02:50.033569 (+ 17225us) server_negotiation.cc:206] Beginning negotiation
0806 03:02:50.033576 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 03:02:50.033604 (+ 28us) server_negotiation.cc:407] Connection header received
0806 03:02:50.033712 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:02:50.033717 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:02:50.033816 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:02:50.039020 (+ 5204us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:02:50.039400 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:50.041427 (+ 2027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:50.043384 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:50.044107 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:50.044261 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:02:50.051825 (+ 7564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:02:50.051833 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:02:50.051838 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:02:50.051878 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:02:50.057337 (+ 5459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:02:50.058262 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:02:50.058268 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:02:50.058271 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:02:50.058462 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:02:50.059093 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:02:50.059097 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:02:50.059100 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:02:50.059514 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:02:50.059722 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0806 03:02:50.060256 (+ 534us) server_negotiation.cc:299] Negotiation successful
0806 03:02:50.060518 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17044,"thread_start_us":17732,"threads_started":1}
I20260806 03:02:50.061148 13615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:50.015260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.252:34107 (local address 127.0.0.1:47116)
0806 03:02:50.015759 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0806 03:02:50.015779 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 03:02:50.016397 (+ 618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:02:50.037877 (+ 21480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:02:50.037883 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:02:50.038516 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:02:50.038814 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:02:50.038833 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:02:50.041628 (+ 2795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:02:50.041634 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:02:50.043186 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:02:50.043207 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:02:50.044307 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:02:50.044314 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:02:50.044635 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:02:50.045338 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:02:50.045367 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:02:50.051469 (+ 6102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:02:50.057495 (+ 6026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:02:50.057501 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:02:50.057505 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:02:50.058097 (+ 592us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:02:50.058633 (+ 536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:02:50.058637 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:02:50.058641 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:02:50.058826 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:02:50.059837 (+ 1011us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:02:50.059842 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:02:50.059980 (+ 138us) client_negotiation.cc:769] Sending connection context
0806 03:02:50.060105 (+ 125us) client_negotiation.cc:240] Negotiation successful
0806 03:02:50.060557 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364}
I20260806 03:02:50.063159 13691 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3af78c2f583343749f10775295433664 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "660bad56e7444e96bff441188d6421f1" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 38641 } } peers { permanent_uuid: "c29e78f76c6f4116998c665614de9e90" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 37437 } } peers { permanent_uuid: "3af78c2f583343749f10775295433664" member_type: VOTER last_known_addr { host: "127.31.74.252" port: 34107 } }
I20260806 03:02:50.063544 13691 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3af78c2f583343749f10775295433664 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:02:50.063678 13691 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3af78c2f583343749f10775295433664 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3af78c2f583343749f10775295433664, State: Initialized, Role: FOLLOWER
I20260806 03:02:50.064343 13691 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3af78c2f583343749f10775295433664 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "660bad56e7444e96bff441188d6421f1" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 38641 } } peers { permanent_uuid: "c29e78f76c6f4116998c665614de9e90" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 37437 } } peers { permanent_uuid: "3af78c2f583343749f10775295433664" member_type: VOTER last_known_addr { host: "127.31.74.252" port: 34107 } }
I20260806 03:02:50.067040 13710 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3af78c2f583343749f10775295433664 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "660bad56e7444e96bff441188d6421f1" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 38641 } } peers { permanent_uuid: "c29e78f76c6f4116998c665614de9e90" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 37437 } } peers { permanent_uuid: "3af78c2f583343749f10775295433664" member_type: VOTER last_known_addr { host: "127.31.74.252" port: 34107 } } }
I20260806 03:02:50.067498 13710 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3af78c2f583343749f10775295433664 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 03:02:50.067679 13612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c29e78f76c6f4116998c665614de9e90: Bootstrap starting.
I20260806 03:02:50.068429 13691 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3af78c2f583343749f10775295433664 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:02:50.071841 13612 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c29e78f76c6f4116998c665614de9e90: Neither blocks nor log segments found. Creating new log.
I20260806 03:02:50.073323 13612 log.cc:824] T 00000000000000000000000000000000 P c29e78f76c6f4116998c665614de9e90: Log is configured to *not* fsync() on all Append() calls
I20260806 03:02:50.078115 13612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c29e78f76c6f4116998c665614de9e90: No bootstrap required, opened a new log
I20260806 03:02:50.081655 13536 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.74.252" port: 34107 } attempt: 5
I20260806 03:02:50.090963 13612 raft_consensus.cc:348] T 00000000000000000000000000000000 P c29e78f76c6f4116998c665614de9e90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "660bad56e7444e96bff441188d6421f1" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 38641 } } peers { permanent_uuid: "c29e78f76c6f4116998c665614de9e90" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 37437 } } peers { permanent_uuid: "3af78c2f583343749f10775295433664" member_type: VOTER last_known_addr { host: "127.31.74.252" port: 34107 } }
I20260806 03:02:50.091655 13612 raft_consensus.cc:374] T 00000000000000000000000000000000 P c29e78f76c6f4116998c665614de9e90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:02:50.092023 13612 raft_consensus.cc:729] T 00000000000000000000000000000000 P c29e78f76c6f4116998c665614de9e90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c29e78f76c6f4116998c665614de9e90, State: Initialized, Role: FOLLOWER
W20260806 03:02:50.092571 13723 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3af78c2f583343749f10775295433664: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 03:02:50.092730 13723 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 03:02:50.093410 13612 consensus_queue.cc:260] T 00000000000000000000000000000000 P c29e78f76c6f4116998c665614de9e90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "660bad56e7444e96bff441188d6421f1" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 38641 } } peers { permanent_uuid: "c29e78f76c6f4116998c665614de9e90" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 37437 } } peers { permanent_uuid: "3af78c2f583343749f10775295433664" member_type: VOTER last_known_addr { host: "127.31.74.252" port: 34107 } }
I20260806 03:02:50.078088 13621 master_runner.cc:186] Error getting master registration for 127.31.74.254:38641: OK, instance_id { permanent_uuid: "660bad56e7444e96bff441188d6421f1" instance_seqno: 1785985368636009 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 03:02:50.095875 13621 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 03:02:50.096675 13725 sys_catalog.cc:455] T 00000000000000000000000000000000 P c29e78f76c6f4116998c665614de9e90 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "660bad56e7444e96bff441188d6421f1" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 38641 } } peers { permanent_uuid: "c29e78f76c6f4116998c665614de9e90" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 37437 } } peers { permanent_uuid: "3af78c2f583343749f10775295433664" member_type: VOTER last_known_addr { host: "127.31.74.252" port: 34107 } } }
I20260806 03:02:50.097268 13725 sys_catalog.cc:458] T 00000000000000000000000000000000 P c29e78f76c6f4116998c665614de9e90 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 03:02:50.098248 13612 sys_catalog.cc:564] T 00000000000000000000000000000000 P c29e78f76c6f4116998c665614de9e90 [sys.catalog]: configured and running, proceeding with master startup.
Aug 06 03:02:50 dist-test-slave-dhph krb5kdc[13453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985368, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.254@KRBTEST.COM for kudu/127.31.74.252@KRBTEST.COM
I20260806 03:02:50.108909 13539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:50.082301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.252:34107 (local address 127.0.0.1:47122)
0806 03:02:50.082501 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0806 03:02:50.082518 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 03:02:50.082669 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:02:50.085546 (+ 2877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:02:50.085556 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:02:50.086002 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:02:50.086197 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:02:50.086208 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:02:50.087902 (+ 1694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:02:50.087910 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:02:50.088936 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:02:50.088951 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:02:50.089778 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:02:50.089785 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:02:50.089998 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:02:50.090801 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:02:50.090828 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:02:50.099570 (+ 8742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:02:50.105547 (+ 5977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:02:50.105553 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:02:50.105558 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:02:50.106135 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:02:50.106774 (+ 639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:02:50.106779 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:02:50.106783 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:02:50.106960 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:02:50.107775 (+ 815us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:02:50.107780 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:02:50.107958 (+ 178us) client_negotiation.cc:769] Sending connection context
0806 03:02:50.108121 (+ 163us) client_negotiation.cc:240] Negotiation successful
0806 03:02:50.108436 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260806 03:02:50.109112 13708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:50.083783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47122 (local address 127.31.74.252:34107)
0806 03:02:50.084229 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0806 03:02:50.084242 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 03:02:50.084294 (+ 52us) server_negotiation.cc:407] Connection header received
0806 03:02:50.084406 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:02:50.084412 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:02:50.084718 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:02:50.084932 (+ 214us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:02:50.086415 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:50.087721 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:50.089107 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:50.089582 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:50.089754 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:02:50.099907 (+ 10153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:02:50.099917 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:02:50.099926 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:02:50.099984 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:02:50.105329 (+ 5345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:02:50.106287 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:02:50.106296 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:02:50.106303 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:02:50.106609 (+ 306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:02:50.107089 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:02:50.107097 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:02:50.107103 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:02:50.107609 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:02:50.107789 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0806 03:02:50.108254 (+ 465us) server_negotiation.cc:299] Negotiation successful
0806 03:02:50.108559 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260806 03:02:50.117550 13536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 660bad56e7444e96bff441188d6421f1: Bootstrap starting.
I20260806 03:02:50.122851 13536 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 660bad56e7444e96bff441188d6421f1: Neither blocks nor log segments found. Creating new log.
I20260806 03:02:50.124392 13536 log.cc:824] T 00000000000000000000000000000000 P 660bad56e7444e96bff441188d6421f1: Log is configured to *not* fsync() on all Append() calls
W20260806 03:02:50.132723 13644 tablet.cc:2369] T 00000000000000000000000000000000 P 3af78c2f583343749f10775295433664: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 03:02:50.134451 13536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 660bad56e7444e96bff441188d6421f1: No bootstrap required, opened a new log
I20260806 03:02:50.123500 13543 master_runner.cc:186] Error getting master registration for 127.31.74.254:38641: OK, instance_id { permanent_uuid: "660bad56e7444e96bff441188d6421f1" instance_seqno: 1785985368636009 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 03:02:50.137573 13543 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 03:02:50.144174 13536 raft_consensus.cc:348] T 00000000000000000000000000000000 P 660bad56e7444e96bff441188d6421f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "660bad56e7444e96bff441188d6421f1" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 38641 } } peers { permanent_uuid: "c29e78f76c6f4116998c665614de9e90" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 37437 } } peers { permanent_uuid: "3af78c2f583343749f10775295433664" member_type: VOTER last_known_addr { host: "127.31.74.252" port: 34107 } }
I20260806 03:02:50.144517 13536 raft_consensus.cc:374] T 00000000000000000000000000000000 P 660bad56e7444e96bff441188d6421f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:02:50.144672 13536 raft_consensus.cc:729] T 00000000000000000000000000000000 P 660bad56e7444e96bff441188d6421f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 660bad56e7444e96bff441188d6421f1, State: Initialized, Role: FOLLOWER
W20260806 03:02:50.145853 13737 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c29e78f76c6f4116998c665614de9e90: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260806 03:02:50.145464 13536 consensus_queue.cc:260] T 00000000000000000000000000000000 P 660bad56e7444e96bff441188d6421f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "660bad56e7444e96bff441188d6421f1" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 38641 } } peers { permanent_uuid: "c29e78f76c6f4116998c665614de9e90" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 37437 } } peers { permanent_uuid: "3af78c2f583343749f10775295433664" member_type: VOTER last_known_addr { host: "127.31.74.252" port: 34107 } }
W20260806 03:02:50.145980 13737 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 03:02:50.157215 13536 sys_catalog.cc:564] T 00000000000000000000000000000000 P 660bad56e7444e96bff441188d6421f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:02:50.155390 13741 sys_catalog.cc:455] T 00000000000000000000000000000000 P 660bad56e7444e96bff441188d6421f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "660bad56e7444e96bff441188d6421f1" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 38641 } } peers { permanent_uuid: "c29e78f76c6f4116998c665614de9e90" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 37437 } } peers { permanent_uuid: "3af78c2f583343749f10775295433664" member_type: VOTER last_known_addr { host: "127.31.74.252" port: 34107 } } }
I20260806 03:02:50.159648 13741 sys_catalog.cc:458] T 00000000000000000000000000000000 P 660bad56e7444e96bff441188d6421f1 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 03:02:50.186282 13467 master_runner.cc:263] No leader master found from master 660bad56e7444e96bff441188d6421f1
I20260806 03:02:50.186497 13467 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260806 03:02:50.195411 13753 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 660bad56e7444e96bff441188d6421f1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 03:02:50.195858 13753 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 03:02:50.242089 13741 raft_consensus.cc:482] T 00000000000000000000000000000000 P 660bad56e7444e96bff441188d6421f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:02:50.242784 13741 raft_consensus.cc:504] T 00000000000000000000000000000000 P 660bad56e7444e96bff441188d6421f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "660bad56e7444e96bff441188d6421f1" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 38641 } } peers { permanent_uuid: "c29e78f76c6f4116998c665614de9e90" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 37437 } } peers { permanent_uuid: "3af78c2f583343749f10775295433664" member_type: VOTER last_known_addr { host: "127.31.74.252" port: 34107 } }
I20260806 03:02:50.244614 13741 leader_election.cc:288] T 00000000000000000000000000000000 P 660bad56e7444e96bff441188d6421f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c29e78f76c6f4116998c665614de9e90 (127.31.74.253:37437), 3af78c2f583343749f10775295433664 (127.31.74.252:34107)
I20260806 03:02:50.247452 13666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "660bad56e7444e96bff441188d6421f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3af78c2f583343749f10775295433664" is_pre_election: true
I20260806 03:02:50.248220 13666 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3af78c2f583343749f10775295433664 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 660bad56e7444e96bff441188d6421f1 in term 0.
W20260806 03:02:50.249202 13565 tablet.cc:2369] T 00000000000000000000000000000000 P c29e78f76c6f4116998c665614de9e90: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 03:02:50.249135 13486 leader_election.cc:302] T 00000000000000000000000000000000 P 660bad56e7444e96bff441188d6421f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3af78c2f583343749f10775295433664, 660bad56e7444e96bff441188d6421f1; no voters:
I20260806 03:02:50.246207 13586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "660bad56e7444e96bff441188d6421f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c29e78f76c6f4116998c665614de9e90" is_pre_election: true
I20260806 03:02:50.250164 13586 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c29e78f76c6f4116998c665614de9e90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 660bad56e7444e96bff441188d6421f1 in term 0.
I20260806 03:02:50.251287 13741 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 660bad56e7444e96bff441188d6421f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:02:50.251467 13741 raft_consensus.cc:482] T 00000000000000000000000000000000 P 660bad56e7444e96bff441188d6421f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:02:50.252213 13741 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 660bad56e7444e96bff441188d6421f1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:02:50.257443 13741 raft_consensus.cc:504] T 00000000000000000000000000000000 P 660bad56e7444e96bff441188d6421f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "660bad56e7444e96bff441188d6421f1" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 38641 } } peers { permanent_uuid: "c29e78f76c6f4116998c665614de9e90" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 37437 } } peers { permanent_uuid: "3af78c2f583343749f10775295433664" member_type: VOTER last_known_addr { host: "127.31.74.252" port: 34107 } }
I20260806 03:02:50.265565 13586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "660bad56e7444e96bff441188d6421f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c29e78f76c6f4116998c665614de9e90"
I20260806 03:02:50.265863 13586 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c29e78f76c6f4116998c665614de9e90 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:02:50.280459 13586 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c29e78f76c6f4116998c665614de9e90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 660bad56e7444e96bff441188d6421f1 in term 1.
I20260806 03:02:50.281462 13486 leader_election.cc:302] T 00000000000000000000000000000000 P 660bad56e7444e96bff441188d6421f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 660bad56e7444e96bff441188d6421f1, c29e78f76c6f4116998c665614de9e90; no voters:
I20260806 03:02:50.291684 13741 leader_election.cc:288] T 00000000000000000000000000000000 P 660bad56e7444e96bff441188d6421f1 [CANDIDATE]: Term 1 election: Requested vote from peers c29e78f76c6f4116998c665614de9e90 (127.31.74.253:37437), 3af78c2f583343749f10775295433664 (127.31.74.252:34107)
I20260806 03:02:50.292124 13741 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 660bad56e7444e96bff441188d6421f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:02:50.292750 13741 raft_consensus.cc:686] T 00000000000000000000000000000000 P 660bad56e7444e96bff441188d6421f1 [term 1 LEADER]: Becoming Leader. State: Replica: 660bad56e7444e96bff441188d6421f1, State: Running, Role: LEADER
I20260806 03:02:50.295450 13741 consensus_queue.cc:237] T 00000000000000000000000000000000 P 660bad56e7444e96bff441188d6421f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "660bad56e7444e96bff441188d6421f1" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 38641 } } peers { permanent_uuid: "c29e78f76c6f4116998c665614de9e90" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 37437 } } peers { permanent_uuid: "3af78c2f583343749f10775295433664" member_type: VOTER last_known_addr { host: "127.31.74.252" port: 34107 } }
I20260806 03:02:50.301266 13666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "660bad56e7444e96bff441188d6421f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3af78c2f583343749f10775295433664"
I20260806 03:02:50.301546 13666 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3af78c2f583343749f10775295433664 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:02:50.306038 13666 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3af78c2f583343749f10775295433664 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 660bad56e7444e96bff441188d6421f1 in term 1.
I20260806 03:02:50.328473 13755 sys_catalog.cc:455] T 00000000000000000000000000000000 P 660bad56e7444e96bff441188d6421f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 660bad56e7444e96bff441188d6421f1. Latest consensus state: current_term: 1 leader_uuid: "660bad56e7444e96bff441188d6421f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "660bad56e7444e96bff441188d6421f1" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 38641 } } peers { permanent_uuid: "c29e78f76c6f4116998c665614de9e90" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 37437 } } peers { permanent_uuid: "3af78c2f583343749f10775295433664" member_type: VOTER last_known_addr { host: "127.31.74.252" port: 34107 } } }
I20260806 03:02:50.329504 13755 sys_catalog.cc:458] T 00000000000000000000000000000000 P 660bad56e7444e96bff441188d6421f1 [sys.catalog]: This master's current role is: LEADER
I20260806 03:02:50.343506 13757 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:02:50.346817 13757 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:02:50.369415 13666 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3af78c2f583343749f10775295433664 [term 1 FOLLOWER]: Refusing update from remote peer 660bad56e7444e96bff441188d6421f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:02:50.370914 13755 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 660bad56e7444e96bff441188d6421f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3af78c2f583343749f10775295433664" member_type: VOTER last_known_addr { host: "127.31.74.252" port: 34107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 03:02:50.372367 13488 tablet.cc:2369] T 00000000000000000000000000000000 P 660bad56e7444e96bff441188d6421f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 03:02:50.383014 13586 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c29e78f76c6f4116998c665614de9e90 [term 1 FOLLOWER]: Refusing update from remote peer 660bad56e7444e96bff441188d6421f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:02:50.384087 13755 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 660bad56e7444e96bff441188d6421f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c29e78f76c6f4116998c665614de9e90" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 37437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:02:50.420207 13725 sys_catalog.cc:455] T 00000000000000000000000000000000 P c29e78f76c6f4116998c665614de9e90 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 660bad56e7444e96bff441188d6421f1. Latest consensus state: current_term: 1 leader_uuid: "660bad56e7444e96bff441188d6421f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "660bad56e7444e96bff441188d6421f1" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 38641 } } peers { permanent_uuid: "c29e78f76c6f4116998c665614de9e90" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 37437 } } peers { permanent_uuid: "3af78c2f583343749f10775295433664" member_type: VOTER last_known_addr { host: "127.31.74.252" port: 34107 } } }
I20260806 03:02:50.420578 13725 sys_catalog.cc:458] T 00000000000000000000000000000000 P c29e78f76c6f4116998c665614de9e90 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 03:02:50.425143 13758 mvcc.cc:205] Tried to move back new op lower bound from 7315396077016834048 to 7315396076795772928. Current Snapshot: MvccSnapshot[applied={T|T < 7315396077016834048}]
I20260806 03:02:50.427075 13755 sys_catalog.cc:455] T 00000000000000000000000000000000 P 660bad56e7444e96bff441188d6421f1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "660bad56e7444e96bff441188d6421f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "660bad56e7444e96bff441188d6421f1" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 38641 } } peers { permanent_uuid: "c29e78f76c6f4116998c665614de9e90" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 37437 } } peers { permanent_uuid: "3af78c2f583343749f10775295433664" member_type: VOTER last_known_addr { host: "127.31.74.252" port: 34107 } } }
I20260806 03:02:50.427584 13755 sys_catalog.cc:458] T 00000000000000000000000000000000 P 660bad56e7444e96bff441188d6421f1 [sys.catalog]: This master's current role is: LEADER
I20260806 03:02:50.430843 13761 mvcc.cc:205] Tried to move back new op lower bound from 7315396077016834048 to 7315396076795772928. Current Snapshot: MvccSnapshot[applied={T|T < 7315396077016834048}]
I20260806 03:02:50.431468 13757 catalog_manager.cc:1347] Generated new cluster ID: 85d50206d4544652a439d1bccbe3341a
I20260806 03:02:50.431581 13757 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:02:50.435950 13710 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3af78c2f583343749f10775295433664 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 660bad56e7444e96bff441188d6421f1. Latest consensus state: current_term: 1 leader_uuid: "660bad56e7444e96bff441188d6421f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "660bad56e7444e96bff441188d6421f1" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 38641 } } peers { permanent_uuid: "c29e78f76c6f4116998c665614de9e90" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 37437 } } peers { permanent_uuid: "3af78c2f583343749f10775295433664" member_type: VOTER last_known_addr { host: "127.31.74.252" port: 34107 } } }
I20260806 03:02:50.436348 13710 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3af78c2f583343749f10775295433664 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 03:02:50.443359 13725 sys_catalog.cc:455] T 00000000000000000000000000000000 P c29e78f76c6f4116998c665614de9e90 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "660bad56e7444e96bff441188d6421f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "660bad56e7444e96bff441188d6421f1" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 38641 } } peers { permanent_uuid: "c29e78f76c6f4116998c665614de9e90" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 37437 } } peers { permanent_uuid: "3af78c2f583343749f10775295433664" member_type: VOTER last_known_addr { host: "127.31.74.252" port: 34107 } } }
I20260806 03:02:50.443709 13725 sys_catalog.cc:458] T 00000000000000000000000000000000 P c29e78f76c6f4116998c665614de9e90 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 03:02:50.454011 13755 sys_catalog.cc:455] T 00000000000000000000000000000000 P 660bad56e7444e96bff441188d6421f1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "660bad56e7444e96bff441188d6421f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "660bad56e7444e96bff441188d6421f1" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 38641 } } peers { permanent_uuid: "c29e78f76c6f4116998c665614de9e90" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 37437 } } peers { permanent_uuid: "3af78c2f583343749f10775295433664" member_type: VOTER last_known_addr { host: "127.31.74.252" port: 34107 } } }
I20260806 03:02:50.454358 13755 sys_catalog.cc:458] T 00000000000000000000000000000000 P 660bad56e7444e96bff441188d6421f1 [sys.catalog]: This master's current role is: LEADER
I20260806 03:02:50.460062 13759 mvcc.cc:205] Tried to move back new op lower bound from 7315396077016834048 to 7315396076795772928. Current Snapshot: MvccSnapshot[applied={T|T < 7315396077016834048}]
I20260806 03:02:50.471095 13710 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3af78c2f583343749f10775295433664 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "660bad56e7444e96bff441188d6421f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "660bad56e7444e96bff441188d6421f1" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 38641 } } peers { permanent_uuid: "c29e78f76c6f4116998c665614de9e90" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 37437 } } peers { permanent_uuid: "3af78c2f583343749f10775295433664" member_type: VOTER last_known_addr { host: "127.31.74.252" port: 34107 } } }
I20260806 03:02:50.471426 13710 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3af78c2f583343749f10775295433664 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 03:02:50.560068 13757 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:02:50.563773 13757 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:02:50.629647 13757 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 660bad56e7444e96bff441188d6421f1: Generated new TSK 0
I20260806 03:02:50.631342 13757 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 03:02:51.095326 13723 catalog_manager.cc:1259] Loaded cluster ID: 85d50206d4544652a439d1bccbe3341a
I20260806 03:02:51.095449 13723 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3af78c2f583343749f10775295433664: loading cluster ID for follower catalog manager: success
I20260806 03:02:51.098848 13723 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3af78c2f583343749f10775295433664: acquiring CA information for follower catalog manager: success
I20260806 03:02:51.100978 13723 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3af78c2f583343749f10775295433664: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 03:02:51.148495 13737 catalog_manager.cc:1259] Loaded cluster ID: 85d50206d4544652a439d1bccbe3341a
I20260806 03:02:51.148627 13737 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c29e78f76c6f4116998c665614de9e90: loading cluster ID for follower catalog manager: success
I20260806 03:02:51.151851 13737 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c29e78f76c6f4116998c665614de9e90: acquiring CA information for follower catalog manager: success
I20260806 03:02:51.154073 13737 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c29e78f76c6f4116998c665614de9e90: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 03:02:51.285163 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 13467
I20260806 03:02:51.286747 13531 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 03:02:51.408957 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 13467
I20260806 03:02:51.435806 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 13543
I20260806 03:02:51.437441 13607 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 03:02:51.557062 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 13543
I20260806 03:02:51.582232 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 13621
I20260806 03:02:51.583562 13686 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 03:02:51.703431 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 13621
2026-08-06T03:02:51Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (5246 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785985105255475-32043-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 03:02:51 dist-test-slave-dhph krb5kdc[13777](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 03:02:51 dist-test-slave-dhph krb5kdc[13777](info): set up 2 sockets
Aug 06 03:02:51 dist-test-slave-dhph krb5kdc[13777](info): commencing operation
krb5kdc: starting...
W20260806 03:02:52.981590 32043 mini_kdc.cc:121] Time spent starting KDC: real 1.229s user 0.003s sys 0.044s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785985105255475-32043-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785985105255475-32043-0/krb5kdc/test-user.keytab.
Aug 06 03:02:53 dist-test-slave-dhph krb5kdc[13777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985373, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T03:02:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:02:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.74.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab.
Entry for principal kudu/127.31.74.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab.
I20260806 03:02:53.335959 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785985105255475-32043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785985105255475-32043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785985105255475-32043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:36897
--webserver_interface=127.31.74.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab
--principal=kudu/127.31.74.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:39917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.74.254:36897
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:02:53.738832 13790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:02:53.739214 13790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:02:53.739411 13790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:02:53.748768 13790 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:02:53.748921 13790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:02:53.749023 13790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:02:53.749133 13790 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:02:53.749230 13790 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:02:53.761768 13790 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:39917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785985105255475-32043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785985105255475-32043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.74.254:36897
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab
--principal=kudu/127.31.74.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:36897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.74.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785985105255475-32043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:02:53.763350 13790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:02:53.764809 13790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:02:53.777768 13795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:02:53.778041 13796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:02:53.777873 13798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:02:53.780293 13790 server_base.cc:1034] running on GCE node
I20260806 03:02:53.781719 13790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:02:53.784150 13790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:02:53.785455 13790 hybrid_clock.cc:648] HybridClock initialized: now 1785985373785383 us; error 61 us; skew 500 ppm
I20260806 03:02:53.785871 13790 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 03:02:53 dist-test-slave-dhph krb5kdc[13777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985373, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:02:53.789667 13790 init.cc:377] Logged in from keytab as kudu/127.31.74.254@KRBTEST.COM (short username kudu)
I20260806 03:02:53.791571 13790 webserver.cc:460] Webserver started at http://127.31.74.254:37571/ using document root <none> and password file <none>
I20260806 03:02:53.792411 13790 fs_manager.cc:359] Metadata directory not provided
I20260806 03:02:53.792563 13790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:02:53.792893 13790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:02:53.796221 13790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785985105255475-32043-0/minicluster-data/master-0/data/instance:
uuid: "fedad4f29bf54c3fa4764fc6570bebd3"
format_stamp: "Formatted at 2026-08-06 03:02:53 on dist-test-slave-dhph"
server_key: "100a82075c18cb77293ec521e13e3a63"
server_key_iv: "b7d351a021d61886bb56d5013982fb0f"
server_key_version: "encryptionkey@0"
I20260806 03:02:53.797008 13790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785985105255475-32043-0/minicluster-data/master-0/wal/instance:
uuid: "fedad4f29bf54c3fa4764fc6570bebd3"
format_stamp: "Formatted at 2026-08-06 03:02:53 on dist-test-slave-dhph"
server_key: "100a82075c18cb77293ec521e13e3a63"
server_key_iv: "b7d351a021d61886bb56d5013982fb0f"
server_key_version: "encryptionkey@0"
I20260806 03:02:53.802717 13790 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260806 03:02:53.806931 13805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:02:53.808461 13790 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 03:02:53.808686 13790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785985105255475-32043-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785985105255475-32043-0/minicluster-data/master-0/wal
uuid: "fedad4f29bf54c3fa4764fc6570bebd3"
format_stamp: "Formatted at 2026-08-06 03:02:53 on dist-test-slave-dhph"
server_key: "100a82075c18cb77293ec521e13e3a63"
server_key_iv: "b7d351a021d61886bb56d5013982fb0f"
server_key_version: "encryptionkey@0"
I20260806 03:02:53.808920 13790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785985105255475-32043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785985105255475-32043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785985105255475-32043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:02:53.837553 13790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:02:53.842875 13790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:02:53.843322 13790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:02:53.874241 13790 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.254:36897
I20260806 03:02:53.874274 13858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.254:36897 every 8 connection(s)
I20260806 03:02:53.876616 13790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
I20260806 03:02:53.882464 13859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:02:53.886289 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 13790
I20260806 03:02:53.886656 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785985105255475-32043-0/minicluster-data/master-0/wal/instance
I20260806 03:02:53.887640 32043 external_mini_cluster.cc:1428] Setting key 3a20a82d7632e15d0314ef0bcb141049
I20260806 03:02:53.909857 13859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fedad4f29bf54c3fa4764fc6570bebd3: Bootstrap starting.
Aug 06 03:02:53 dist-test-slave-dhph krb5kdc[13777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985373, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 03:02:53.915729 13859 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fedad4f29bf54c3fa4764fc6570bebd3: Neither blocks nor log segments found. Creating new log.
I20260806 03:02:53.917181 13859 log.cc:824] T 00000000000000000000000000000000 P fedad4f29bf54c3fa4764fc6570bebd3: Log is configured to *not* fsync() on all Append() calls
I20260806 03:02:53.921662 13859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fedad4f29bf54c3fa4764fc6570bebd3: No bootstrap required, opened a new log
I20260806 03:02:53.930048 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:53.892439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40496 (local address 127.31.74.254:36897)
0806 03:02:53.893770 (+ 1331us) server_negotiation.cc:206] Beginning negotiation
0806 03:02:53.893785 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 03:02:53.893838 (+ 53us) server_negotiation.cc:407] Connection header received
0806 03:02:53.896926 (+ 3088us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:02:53.896981 (+ 55us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:02:53.898351 (+ 1370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:02:53.899018 (+ 667us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:02:53.902969 (+ 3951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:53.905807 (+ 2838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:53.908857 (+ 3050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:53.910420 (+ 1563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:53.911679 (+ 1259us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:02:53.916624 (+ 4945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:02:53.916634 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:02:53.916652 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:02:53.916910 (+ 258us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:02:53.922933 (+ 6023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:02:53.924280 (+ 1347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:02:53.924288 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:02:53.924296 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:02:53.925432 (+ 1136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:02:53.926234 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:02:53.926238 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:02:53.926240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:02:53.926985 (+ 745us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:02:53.927533 (+ 548us) server_negotiation.cc:1035] Waiting for connection context
0806 03:02:53.928113 (+ 580us) server_negotiation.cc:299] Negotiation successful
0806 03:02:53.928684 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1059,"thread_start_us":744,"threads_started":1}
I20260806 03:02:53.931468 13859 raft_consensus.cc:348] T 00000000000000000000000000000000 P fedad4f29bf54c3fa4764fc6570bebd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fedad4f29bf54c3fa4764fc6570bebd3" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36897 } }
I20260806 03:02:53.932065 13859 raft_consensus.cc:374] T 00000000000000000000000000000000 P fedad4f29bf54c3fa4764fc6570bebd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:02:53.932183 13859 raft_consensus.cc:729] T 00000000000000000000000000000000 P fedad4f29bf54c3fa4764fc6570bebd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fedad4f29bf54c3fa4764fc6570bebd3, State: Initialized, Role: FOLLOWER
I20260806 03:02:53.932911 13859 consensus_queue.cc:260] T 00000000000000000000000000000000 P fedad4f29bf54c3fa4764fc6570bebd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fedad4f29bf54c3fa4764fc6570bebd3" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36897 } }
I20260806 03:02:53.933231 13859 raft_consensus.cc:388] T 00000000000000000000000000000000 P fedad4f29bf54c3fa4764fc6570bebd3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:02:53.933390 13859 raft_consensus.cc:482] T 00000000000000000000000000000000 P fedad4f29bf54c3fa4764fc6570bebd3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:02:53.933604 13859 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fedad4f29bf54c3fa4764fc6570bebd3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:02:53.935192 32043 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 03:02:53.937039 13859 raft_consensus.cc:504] T 00000000000000000000000000000000 P fedad4f29bf54c3fa4764fc6570bebd3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fedad4f29bf54c3fa4764fc6570bebd3" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36897 } }
I20260806 03:02:53.937690 13859 leader_election.cc:302] T 00000000000000000000000000000000 P fedad4f29bf54c3fa4764fc6570bebd3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fedad4f29bf54c3fa4764fc6570bebd3; no voters:
I20260806 03:02:53.938520 13859 leader_election.cc:288] T 00000000000000000000000000000000 P fedad4f29bf54c3fa4764fc6570bebd3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:02:53.939024 13864 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fedad4f29bf54c3fa4764fc6570bebd3 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:02:53.941504 13859 sys_catalog.cc:564] T 00000000000000000000000000000000 P fedad4f29bf54c3fa4764fc6570bebd3 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:02:53.941977 13864 raft_consensus.cc:686] T 00000000000000000000000000000000 P fedad4f29bf54c3fa4764fc6570bebd3 [term 1 LEADER]: Becoming Leader. State: Replica: fedad4f29bf54c3fa4764fc6570bebd3, State: Running, Role: LEADER
I20260806 03:02:53.942679 13864 consensus_queue.cc:237] T 00000000000000000000000000000000 P fedad4f29bf54c3fa4764fc6570bebd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fedad4f29bf54c3fa4764fc6570bebd3" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36897 } }
I20260806 03:02:53.946444 13866 sys_catalog.cc:455] T 00000000000000000000000000000000 P fedad4f29bf54c3fa4764fc6570bebd3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fedad4f29bf54c3fa4764fc6570bebd3. Latest consensus state: current_term: 1 leader_uuid: "fedad4f29bf54c3fa4764fc6570bebd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fedad4f29bf54c3fa4764fc6570bebd3" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36897 } } }
I20260806 03:02:53.946450 13865 sys_catalog.cc:455] T 00000000000000000000000000000000 P fedad4f29bf54c3fa4764fc6570bebd3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fedad4f29bf54c3fa4764fc6570bebd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fedad4f29bf54c3fa4764fc6570bebd3" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36897 } } }
I20260806 03:02:53.947207 13866 sys_catalog.cc:458] T 00000000000000000000000000000000 P fedad4f29bf54c3fa4764fc6570bebd3 [sys.catalog]: This master's current role is: LEADER
I20260806 03:02:53.947217 13865 sys_catalog.cc:458] T 00000000000000000000000000000000 P fedad4f29bf54c3fa4764fc6570bebd3 [sys.catalog]: This master's current role is: LEADER
I20260806 03:02:53.948269 13875 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:02:53.955232 13875 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:02:53.966089 13875 catalog_manager.cc:1347] Generated new cluster ID: 037f4d39a4464bda82eb58c06e524783
I20260806 03:02:53.966274 13875 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:02:54.024358 13875 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:02:54.026301 13875 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:02:54.043785 13875 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fedad4f29bf54c3fa4764fc6570bebd3: Generated new TSK 0
I20260806 03:02:54.045132 13875 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 03:02:54.102409 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 13790
I20260806 03:02:54.104740 13854 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 03:02:54.221590 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 13790
2026-08-06T03:02:54Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2515 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 03:02:54 dist-test-slave-dhph krb5kdc[13889](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 03:02:54 dist-test-slave-dhph krb5kdc[13889](info): set up 2 sockets
Aug 06 03:02:54 dist-test-slave-dhph krb5kdc[13889](info): commencing operation
krb5kdc: starting...
W20260806 03:02:55.560421 32043 mini_kdc.cc:121] Time spent starting KDC: real 1.293s user 0.004s sys 0.045s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/krb5kdc/test-user.keytab.
Aug 06 03:02:55 dist-test-slave-dhph krb5kdc[13889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985375, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T03:02:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:02:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.74.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab.
Entry for principal kudu/127.31.74.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab.
I20260806 03:02:55.961261 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:33865
--webserver_interface=127.31.74.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab
--principal=kudu/127.31.74.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:44989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.74.254:33865
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:02:56.421523 13903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:02:56.421926 13903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:02:56.422128 13903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:02:56.431241 13903 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:02:56.431381 13903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:02:56.431432 13903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:02:56.431557 13903 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:02:56.431638 13903 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:02:56.446383 13903 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:44989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.74.254:33865
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab
--principal=kudu/127.31.74.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:33865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.74.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:02:56.448143 13903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:02:56.450537 13903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:02:56.467869 13908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:02:56.467785 13909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:02:56.467803 13911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:02:56.469035 13903 server_base.cc:1034] running on GCE node
I20260806 03:02:56.470119 13903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:02:56.472066 13903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:02:56.473371 13903 hybrid_clock.cc:648] HybridClock initialized: now 1785985376473319 us; error 40 us; skew 500 ppm
I20260806 03:02:56.473829 13903 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 03:02:56 dist-test-slave-dhph krb5kdc[13889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985376, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:02:56.478461 13903 init.cc:377] Logged in from keytab as kudu/127.31.74.254@KRBTEST.COM (short username kudu)
I20260806 03:02:56.480583 13903 webserver.cc:460] Webserver started at http://127.31.74.254:39355/ using document root <none> and password file <none>
I20260806 03:02:56.481434 13903 fs_manager.cc:359] Metadata directory not provided
I20260806 03:02:56.481612 13903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:02:56.482002 13903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:02:56.485694 13903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/master-0/data/instance:
uuid: "f53133eff48348659b64246f2f8317c5"
format_stamp: "Formatted at 2026-08-06 03:02:56 on dist-test-slave-dhph"
server_key: "245a2d316f63d43e223222235a5005cc"
server_key_iv: "b2b761d8fb8444f5ee18e31d55c49e36"
server_key_version: "encryptionkey@0"
I20260806 03:02:56.487470 13903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/master-0/wal/instance:
uuid: "f53133eff48348659b64246f2f8317c5"
format_stamp: "Formatted at 2026-08-06 03:02:56 on dist-test-slave-dhph"
server_key: "245a2d316f63d43e223222235a5005cc"
server_key_iv: "b2b761d8fb8444f5ee18e31d55c49e36"
server_key_version: "encryptionkey@0"
I20260806 03:02:56.494094 13903 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.005s
I20260806 03:02:56.498245 13918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:02:56.500077 13903 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 03:02:56.500335 13903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/master-0/wal
uuid: "f53133eff48348659b64246f2f8317c5"
format_stamp: "Formatted at 2026-08-06 03:02:56 on dist-test-slave-dhph"
server_key: "245a2d316f63d43e223222235a5005cc"
server_key_iv: "b2b761d8fb8444f5ee18e31d55c49e36"
server_key_version: "encryptionkey@0"
I20260806 03:02:56.500592 13903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:02:56.524302 13903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:02:56.531168 13903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:02:56.531550 13903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:02:56.555687 13903 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.254:33865
I20260806 03:02:56.555724 13972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.254:33865 every 8 connection(s)
I20260806 03:02:56.557492 13903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
I20260806 03:02:56.562130 13973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:02:56.563824 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 13903
I20260806 03:02:56.564095 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/master-0/wal/instance
I20260806 03:02:56.564857 32043 external_mini_cluster.cc:1428] Setting key 0e70071b4549fe1408180809707a2fe6
I20260806 03:02:56.577452 13973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f53133eff48348659b64246f2f8317c5: Bootstrap starting.
I20260806 03:02:56.582368 13973 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f53133eff48348659b64246f2f8317c5: Neither blocks nor log segments found. Creating new log.
I20260806 03:02:56.584048 13973 log.cc:824] T 00000000000000000000000000000000 P f53133eff48348659b64246f2f8317c5: Log is configured to *not* fsync() on all Append() calls
Aug 06 03:02:56 dist-test-slave-dhph krb5kdc[13889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985375, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 03:02:56.590065 13973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f53133eff48348659b64246f2f8317c5: No bootstrap required, opened a new log
I20260806 03:02:56.602430 13976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:56.567209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40216 (local address 127.31.74.254:33865)
0806 03:02:56.568523 (+ 1314us) server_negotiation.cc:206] Beginning negotiation
0806 03:02:56.568539 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 03:02:56.568600 (+ 61us) server_negotiation.cc:407] Connection header received
0806 03:02:56.571029 (+ 2429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:02:56.571055 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:02:56.571522 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:02:56.572172 (+ 650us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:02:56.574211 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:56.575992 (+ 1781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:56.578162 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:56.579105 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:56.579284 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:02:56.588125 (+ 8841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:02:56.588135 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:02:56.588154 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:02:56.588202 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:02:56.593678 (+ 5476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:02:56.594955 (+ 1277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:02:56.594969 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:02:56.594982 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:02:56.595402 (+ 420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:02:56.596352 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:02:56.596364 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:02:56.596369 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:02:56.597575 (+ 1206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:02:56.598342 (+ 767us) server_negotiation.cc:1035] Waiting for connection context
0806 03:02:56.599536 (+ 1194us) server_negotiation.cc:299] Negotiation successful
0806 03:02:56.600563 (+ 1027us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":861,"thread_start_us":593,"threads_started":1}
I20260806 03:02:56.604422 13973 raft_consensus.cc:348] T 00000000000000000000000000000000 P f53133eff48348659b64246f2f8317c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f53133eff48348659b64246f2f8317c5" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 33865 } }
I20260806 03:02:56.605386 13973 raft_consensus.cc:374] T 00000000000000000000000000000000 P f53133eff48348659b64246f2f8317c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:02:56.605841 13973 raft_consensus.cc:729] T 00000000000000000000000000000000 P f53133eff48348659b64246f2f8317c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f53133eff48348659b64246f2f8317c5, State: Initialized, Role: FOLLOWER
I20260806 03:02:56.607194 13973 consensus_queue.cc:260] T 00000000000000000000000000000000 P f53133eff48348659b64246f2f8317c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f53133eff48348659b64246f2f8317c5" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 33865 } }
I20260806 03:02:56.607563 13973 raft_consensus.cc:388] T 00000000000000000000000000000000 P f53133eff48348659b64246f2f8317c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:02:56.607715 13973 raft_consensus.cc:482] T 00000000000000000000000000000000 P f53133eff48348659b64246f2f8317c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:02:56.607867 13973 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f53133eff48348659b64246f2f8317c5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:02:56.610374 13973 raft_consensus.cc:504] T 00000000000000000000000000000000 P f53133eff48348659b64246f2f8317c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f53133eff48348659b64246f2f8317c5" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 33865 } }
I20260806 03:02:56.611142 13973 leader_election.cc:302] T 00000000000000000000000000000000 P f53133eff48348659b64246f2f8317c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f53133eff48348659b64246f2f8317c5; no voters:
I20260806 03:02:56.612222 13973 leader_election.cc:288] T 00000000000000000000000000000000 P f53133eff48348659b64246f2f8317c5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:02:56.612651 13978 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f53133eff48348659b64246f2f8317c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:02:56.615754 13978 raft_consensus.cc:686] T 00000000000000000000000000000000 P f53133eff48348659b64246f2f8317c5 [term 1 LEADER]: Becoming Leader. State: Replica: f53133eff48348659b64246f2f8317c5, State: Running, Role: LEADER
I20260806 03:02:56.616397 13978 consensus_queue.cc:237] T 00000000000000000000000000000000 P f53133eff48348659b64246f2f8317c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f53133eff48348659b64246f2f8317c5" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 33865 } }
I20260806 03:02:56.617003 13973 sys_catalog.cc:564] T 00000000000000000000000000000000 P f53133eff48348659b64246f2f8317c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:02:56.619985 13979 sys_catalog.cc:455] T 00000000000000000000000000000000 P f53133eff48348659b64246f2f8317c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f53133eff48348659b64246f2f8317c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f53133eff48348659b64246f2f8317c5" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 33865 } } }
I20260806 03:02:56.621454 13979 sys_catalog.cc:458] T 00000000000000000000000000000000 P f53133eff48348659b64246f2f8317c5 [sys.catalog]: This master's current role is: LEADER
I20260806 03:02:56.623289 13980 sys_catalog.cc:455] T 00000000000000000000000000000000 P f53133eff48348659b64246f2f8317c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f53133eff48348659b64246f2f8317c5. Latest consensus state: current_term: 1 leader_uuid: "f53133eff48348659b64246f2f8317c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f53133eff48348659b64246f2f8317c5" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 33865 } } }
I20260806 03:02:56.623639 13980 sys_catalog.cc:458] T 00000000000000000000000000000000 P f53133eff48348659b64246f2f8317c5 [sys.catalog]: This master's current role is: LEADER
I20260806 03:02:56.624313 13986 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:02:56.632942 13986 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:02:56.646836 13986 catalog_manager.cc:1347] Generated new cluster ID: 2cd1e9d5e7e44de1b93b1f61ecaf923a
I20260806 03:02:56.647248 13986 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:02:56.694424 13986 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:02:56.697012 13986 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:02:56.712370 13986 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f53133eff48348659b64246f2f8317c5: Generated new TSK 0
I20260806 03:02:56.713912 13986 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.74.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.193.keytab.
Entry for principal kudu/127.31.74.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.193.keytab.
W20260806 03:02:56.824826 32043 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.31.74.193: real 0.102s user 0.000s sys 0.028s
I20260806 03:02:56.826177 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.193:0
--local_ip_for_outbound_sockets=127.31.74.193
--webserver_interface=127.31.74.193
--webserver_port=0
--tserver_master_addrs=127.31.74.254:33865
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.193.keytab
--principal=kudu/127.31.74.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:44989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:02:57.265218 14000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:02:57.265638 14000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:02:57.265791 14000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:02:57.275228 14000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:02:57.275396 14000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:02:57.275538 14000 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.193
I20260806 03:02:57.287360 14000 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:44989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.193.keytab
--principal=kudu/127.31.74.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.74.193
--webserver_port=0
--tserver_master_addrs=127.31.74.254:33865
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.193
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:02:57.288808 14000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:02:57.290397 14000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:02:57.305640 14006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:02:57.305719 14008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:02:57.305680 14005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:02:57.306742 14000 server_base.cc:1034] running on GCE node
I20260806 03:02:57.307618 14000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:02:57.308905 14000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:02:57.310231 14000 hybrid_clock.cc:648] HybridClock initialized: now 1785985377310152 us; error 69 us; skew 500 ppm
I20260806 03:02:57.310745 14000 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 03:02:57 dist-test-slave-dhph krb5kdc[13889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985377, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:02:57.316057 14000 init.cc:377] Logged in from keytab as kudu/127.31.74.193@KRBTEST.COM (short username kudu)
I20260806 03:02:57.318251 14000 webserver.cc:460] Webserver started at http://127.31.74.193:37087/ using document root <none> and password file <none>
I20260806 03:02:57.319401 14000 fs_manager.cc:359] Metadata directory not provided
I20260806 03:02:57.319787 14000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:02:57.320173 14000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:02:57.323876 14000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-0/data/instance:
uuid: "dc3142ac2c8548c48750d63e7eca5961"
format_stamp: "Formatted at 2026-08-06 03:02:57 on dist-test-slave-dhph"
server_key: "60ac3b964d89f11416c30d1e25045229"
server_key_iv: "8bc41061c44f56028dd17211372facb3"
server_key_version: "encryptionkey@0"
I20260806 03:02:57.324815 14000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-0/wal/instance:
uuid: "dc3142ac2c8548c48750d63e7eca5961"
format_stamp: "Formatted at 2026-08-06 03:02:57 on dist-test-slave-dhph"
server_key: "60ac3b964d89f11416c30d1e25045229"
server_key_iv: "8bc41061c44f56028dd17211372facb3"
server_key_version: "encryptionkey@0"
I20260806 03:02:57.332789 14000 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.000s sys 0.008s
I20260806 03:02:57.337299 14017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:02:57.339119 14000 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260806 03:02:57.339449 14000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-0/wal
uuid: "dc3142ac2c8548c48750d63e7eca5961"
format_stamp: "Formatted at 2026-08-06 03:02:57 on dist-test-slave-dhph"
server_key: "60ac3b964d89f11416c30d1e25045229"
server_key_iv: "8bc41061c44f56028dd17211372facb3"
server_key_version: "encryptionkey@0"
I20260806 03:02:57.339704 14000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:02:57.375625 14000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:02:57.381146 14000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:02:57.381484 14000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:02:57.382615 14000 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:02:57.384156 14000 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:02:57.384281 14000 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:02:57.384440 14000 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:02:57.384536 14000 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:02:57.428952 14000 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.193:37831
I20260806 03:02:57.429248 14137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.193:37831 every 8 connection(s)
I20260806 03:02:57.430459 14000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
I20260806 03:02:57.432550 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 14000
I20260806 03:02:57.432817 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-0/wal/instance
I20260806 03:02:57.433758 32043 external_mini_cluster.cc:1428] Setting key 4a8611bc67a3db3e3ce927340f2e7803
Aug 06 03:02:57 dist-test-slave-dhph krb5kdc[13889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985377, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.193@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 03:02:57.459226 14140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:57.434643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.193:50007 (local address 127.31.74.254:33865)
0806 03:02:57.435330 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0806 03:02:57.435337 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 03:02:57.437154 (+ 1817us) server_negotiation.cc:407] Connection header received
0806 03:02:57.439172 (+ 2018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:02:57.439178 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:02:57.439279 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:02:57.439404 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:02:57.441347 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:57.442115 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:57.443838 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:57.444294 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:57.444441 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:02:57.449037 (+ 4596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:02:57.449048 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:02:57.449055 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:02:57.449128 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:02:57.453186 (+ 4058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:02:57.454896 (+ 1710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:02:57.454907 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:02:57.454915 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:02:57.455130 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:02:57.456265 (+ 1135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:02:57.456273 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:02:57.456280 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:02:57.456998 (+ 718us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:02:57.457212 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0806 03:02:57.458282 (+ 1070us) server_negotiation.cc:299] Negotiation successful
0806 03:02:57.458659 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":349,"threads_started":1}
I20260806 03:02:57.459986 14141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:57.435731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:33865 (local address 127.31.74.193:50007)
0806 03:02:57.436985 (+ 1254us) negotiation.cc:107] Waiting for socket to connect
0806 03:02:57.437029 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 03:02:57.438875 (+ 1846us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:02:57.439675 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:02:57.439684 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:02:57.440238 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:02:57.440848 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:02:57.440865 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:02:57.442301 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:02:57.442305 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:02:57.443680 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:02:57.443693 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:02:57.444555 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:02:57.444562 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:02:57.444815 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:02:57.445598 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:02:57.445626 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:02:57.448685 (+ 3059us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:02:57.453421 (+ 4736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:02:57.453430 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:02:57.453448 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:02:57.454472 (+ 1024us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:02:57.455543 (+ 1071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:02:57.455547 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:02:57.455550 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:02:57.456073 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:02:57.457202 (+ 1129us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:02:57.457210 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:02:57.457745 (+ 535us) client_negotiation.cc:769] Sending connection context
0806 03:02:57.458092 (+ 347us) client_negotiation.cc:240] Negotiation successful
0806 03:02:57.458657 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":951,"thread_start_us":706,"threads_started":1}
I20260806 03:02:57.461961 14138 heartbeater.cc:347] Connected to a master server at 127.31.74.254:33865
I20260806 03:02:57.462512 14138 heartbeater.cc:464] Registering TS with master...
I20260806 03:02:57.463470 14138 heartbeater.cc:511] Master 127.31.74.254:33865 requested a full tablet report, sending...
I20260806 03:02:57.466665 13936 ts_manager.cc:194] Registered new tserver with Master: dc3142ac2c8548c48750d63e7eca5961 (127.31.74.193:37831)
I20260806 03:02:57.468528 13936 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.74.193@KRBTEST.COM'} at 127.31.74.193:50007
WARNING: no policy specified for kudu/127.31.74.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.194.keytab.
Entry for principal kudu/127.31.74.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.194.keytab.
I20260806 03:02:57.515971 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.194:0
--local_ip_for_outbound_sockets=127.31.74.194
--webserver_interface=127.31.74.194
--webserver_port=0
--tserver_master_addrs=127.31.74.254:33865
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.194.keytab
--principal=kudu/127.31.74.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:44989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:02:57.921976 14144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:02:57.922312 14144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:02:57.922417 14144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:02:57.931185 14144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:02:57.931324 14144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:02:57.931517 14144 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.194
I20260806 03:02:57.944800 14144 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:44989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.194.keytab
--principal=kudu/127.31.74.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.74.194
--webserver_port=0
--tserver_master_addrs=127.31.74.254:33865
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.194
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:02:57.946964 14144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:02:57.949419 14144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:02:57.967293 14150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:02:57.967206 14152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:02:57.967257 14149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:02:57.968211 14144 server_base.cc:1034] running on GCE node
I20260806 03:02:57.969054 14144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:02:57.970363 14144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:02:57.971660 14144 hybrid_clock.cc:648] HybridClock initialized: now 1785985377971590 us; error 65 us; skew 500 ppm
I20260806 03:02:57.972178 14144 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 03:02:57 dist-test-slave-dhph krb5kdc[13889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985377, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:02:57.976742 14144 init.cc:377] Logged in from keytab as kudu/127.31.74.194@KRBTEST.COM (short username kudu)
I20260806 03:02:57.978691 14144 webserver.cc:460] Webserver started at http://127.31.74.194:42495/ using document root <none> and password file <none>
I20260806 03:02:57.979467 14144 fs_manager.cc:359] Metadata directory not provided
I20260806 03:02:57.979588 14144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:02:57.979872 14144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:02:57.983489 14144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-1/data/instance:
uuid: "bb22bd13bfd8404aa9d02498577cd212"
format_stamp: "Formatted at 2026-08-06 03:02:57 on dist-test-slave-dhph"
server_key: "3afd3b3c8bda75f0f7e6d1aaa9388e51"
server_key_iv: "493b446ae358887437b3a868469f5f6d"
server_key_version: "encryptionkey@0"
I20260806 03:02:57.984422 14144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-1/wal/instance:
uuid: "bb22bd13bfd8404aa9d02498577cd212"
format_stamp: "Formatted at 2026-08-06 03:02:57 on dist-test-slave-dhph"
server_key: "3afd3b3c8bda75f0f7e6d1aaa9388e51"
server_key_iv: "493b446ae358887437b3a868469f5f6d"
server_key_version: "encryptionkey@0"
I20260806 03:02:57.991660 14144 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260806 03:02:57.997574 14159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:02:58.000020 14144 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 03:02:58.000236 14144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-1/wal
uuid: "bb22bd13bfd8404aa9d02498577cd212"
format_stamp: "Formatted at 2026-08-06 03:02:57 on dist-test-slave-dhph"
server_key: "3afd3b3c8bda75f0f7e6d1aaa9388e51"
server_key_iv: "493b446ae358887437b3a868469f5f6d"
server_key_version: "encryptionkey@0"
I20260806 03:02:58.000480 14144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:02:58.040978 14144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:02:58.048184 14144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:02:58.048509 14144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:02:58.049649 14144 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:02:58.051564 14144 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:02:58.051687 14144 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:02:58.051801 14144 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:02:58.051860 14144 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:02:58.095952 14144 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.194:45731
I20260806 03:02:58.095992 14276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.194:45731 every 8 connection(s)
I20260806 03:02:58.097306 14144 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
I20260806 03:02:58.104637 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 14144
I20260806 03:02:58.105172 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-1/wal/instance
I20260806 03:02:58.105964 32043 external_mini_cluster.cc:1428] Setting key 10d71116a1f05fdaddccfb808312a47b
Aug 06 03:02:58 dist-test-slave-dhph krb5kdc[13889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985377, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.194@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 03:02:58.122510 14279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:58.100389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.194:53325 (local address 127.31.74.254:33865)
0806 03:02:58.100958 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0806 03:02:58.100963 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 03:02:58.102105 (+ 1142us) server_negotiation.cc:407] Connection header received
0806 03:02:58.104284 (+ 2179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:02:58.104291 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:02:58.104394 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:02:58.104572 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:02:58.107020 (+ 2448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:58.107824 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:58.109449 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:58.109900 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:58.110056 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:02:58.114936 (+ 4880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:02:58.114942 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:02:58.114945 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:02:58.114996 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:02:58.118798 (+ 3802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:02:58.119747 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:02:58.119752 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:02:58.119755 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:02:58.119911 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:02:58.120407 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:02:58.120417 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:02:58.120420 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:02:58.120863 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:02:58.121053 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0806 03:02:58.121849 (+ 796us) server_negotiation.cc:299] Negotiation successful
0806 03:02:58.122107 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":344,"threads_started":1}
I20260806 03:02:58.123764 14280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:58.100978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:33865 (local address 127.31.74.194:53325)
0806 03:02:58.101886 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0806 03:02:58.101938 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 03:02:58.103962 (+ 2024us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:02:58.105062 (+ 1100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:02:58.105079 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:02:58.106034 (+ 955us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:02:58.106821 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:02:58.106846 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:02:58.107989 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:02:58.107999 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:02:58.109282 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:02:58.109295 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:02:58.110156 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:02:58.110163 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:02:58.110417 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:02:58.111283 (+ 866us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:02:58.111313 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:02:58.114513 (+ 3200us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:02:58.118990 (+ 4477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:02:58.119005 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:02:58.119022 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:02:58.119569 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:02:58.120041 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:02:58.120047 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:02:58.120050 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:02:58.120267 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:02:58.121022 (+ 755us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:02:58.121030 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:02:58.121438 (+ 408us) client_negotiation.cc:769] Sending connection context
0806 03:02:58.121744 (+ 306us) client_negotiation.cc:240] Negotiation successful
0806 03:02:58.122350 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":340,"threads_started":1}
I20260806 03:02:58.125845 14277 heartbeater.cc:347] Connected to a master server at 127.31.74.254:33865
I20260806 03:02:58.126325 14277 heartbeater.cc:464] Registering TS with master...
I20260806 03:02:58.127235 14277 heartbeater.cc:511] Master 127.31.74.254:33865 requested a full tablet report, sending...
I20260806 03:02:58.129168 13936 ts_manager.cc:194] Registered new tserver with Master: bb22bd13bfd8404aa9d02498577cd212 (127.31.74.194:45731)
I20260806 03:02:58.130221 13936 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.74.194@KRBTEST.COM'} at 127.31.74.194:53325
WARNING: no policy specified for kudu/127.31.74.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.195.keytab.
Entry for principal kudu/127.31.74.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.195.keytab.
I20260806 03:02:58.194195 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.195:0
--local_ip_for_outbound_sockets=127.31.74.195
--webserver_interface=127.31.74.195
--webserver_port=0
--tserver_master_addrs=127.31.74.254:33865
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.195.keytab
--principal=kudu/127.31.74.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:44989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 03:02:58.472993 14138 heartbeater.cc:503] Master 127.31.74.254:33865 was elected leader, sending a full tablet report...
W20260806 03:02:58.612818 14283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:02:58.613379 14283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:02:58.613561 14283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:02:58.621904 14283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:02:58.622313 14283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:02:58.622538 14283 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.195
I20260806 03:02:58.634303 14283 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:44989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.195.keytab
--principal=kudu/127.31.74.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.74.195
--webserver_port=0
--tserver_master_addrs=127.31.74.254:33865
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.195
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:02:58.635842 14283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:02:58.637493 14283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:02:58.654551 14289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:02:58.654520 14292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:02:58.654604 14290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:02:58.655584 14283 server_base.cc:1034] running on GCE node
I20260806 03:02:58.656599 14283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:02:58.658313 14283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:02:58.659718 14283 hybrid_clock.cc:648] HybridClock initialized: now 1785985378659643 us; error 73 us; skew 500 ppm
I20260806 03:02:58.660409 14283 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 03:02:58 dist-test-slave-dhph krb5kdc[13889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985378, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:02:58.665416 14283 init.cc:377] Logged in from keytab as kudu/127.31.74.195@KRBTEST.COM (short username kudu)
I20260806 03:02:58.668098 14283 webserver.cc:460] Webserver started at http://127.31.74.195:35447/ using document root <none> and password file <none>
I20260806 03:02:58.669219 14283 fs_manager.cc:359] Metadata directory not provided
I20260806 03:02:58.669400 14283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:02:58.669750 14283 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:02:58.673355 14283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-2/data/instance:
uuid: "2ee2abdf415246509365cee48c030d36"
format_stamp: "Formatted at 2026-08-06 03:02:58 on dist-test-slave-dhph"
server_key: "ec3cee2c35ef448d2ed30c57459f5885"
server_key_iv: "c75b10fff81f46f9977292ab6a2b8c6e"
server_key_version: "encryptionkey@0"
I20260806 03:02:58.674401 14283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-2/wal/instance:
uuid: "2ee2abdf415246509365cee48c030d36"
format_stamp: "Formatted at 2026-08-06 03:02:58 on dist-test-slave-dhph"
server_key: "ec3cee2c35ef448d2ed30c57459f5885"
server_key_iv: "c75b10fff81f46f9977292ab6a2b8c6e"
server_key_version: "encryptionkey@0"
I20260806 03:02:58.680835 14283 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260806 03:02:58.685751 14299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:02:58.688412 14283 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260806 03:02:58.688678 14283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-2/wal
uuid: "2ee2abdf415246509365cee48c030d36"
format_stamp: "Formatted at 2026-08-06 03:02:58 on dist-test-slave-dhph"
server_key: "ec3cee2c35ef448d2ed30c57459f5885"
server_key_iv: "c75b10fff81f46f9977292ab6a2b8c6e"
server_key_version: "encryptionkey@0"
I20260806 03:02:58.688948 14283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:02:58.731041 14283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:02:58.737149 14283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:02:58.737520 14283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:02:58.738807 14283 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:02:58.741741 14283 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:02:58.741906 14283 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:02:58.742106 14283 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:02:58.742193 14283 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:02:58.786720 14283 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.195:32815
I20260806 03:02:58.786747 14413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.195:32815 every 8 connection(s)
I20260806 03:02:58.788189 14283 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
I20260806 03:02:58.795768 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 14283
I20260806 03:02:58.796032 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-2/wal/instance
I20260806 03:02:58.796707 32043 external_mini_cluster.cc:1428] Setting key c616c4061fc56ea704f9267d6fb572af
Aug 06 03:02:58 dist-test-slave-dhph krb5kdc[13889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985378, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.195@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 03:02:58.819720 14416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:58.791315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.195:33147 (local address 127.31.74.254:33865)
0806 03:02:58.791845 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0806 03:02:58.791850 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 03:02:58.793058 (+ 1208us) server_negotiation.cc:407] Connection header received
0806 03:02:58.795940 (+ 2882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:02:58.795949 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:02:58.796055 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:02:58.796319 (+ 264us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:02:58.800255 (+ 3936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:58.801276 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:58.803881 (+ 2605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:58.804927 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:58.805042 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:02:58.809963 (+ 4921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:02:58.809969 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:02:58.809973 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:02:58.810016 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:02:58.814179 (+ 4163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:02:58.815848 (+ 1669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:02:58.815856 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:02:58.815860 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:02:58.816160 (+ 300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:02:58.817349 (+ 1189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:02:58.817355 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:02:58.817358 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:02:58.817785 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:02:58.818012 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0806 03:02:58.818957 (+ 945us) server_negotiation.cc:299] Negotiation successful
0806 03:02:58.819258 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":263,"threads_started":1}
I20260806 03:02:58.820840 14417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:58.792024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:33865 (local address 127.31.74.195:33147)
0806 03:02:58.792860 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0806 03:02:58.792911 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 03:02:58.795364 (+ 2453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:02:58.796717 (+ 1353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:02:58.796730 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:02:58.797546 (+ 816us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:02:58.799997 (+ 2451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:02:58.800030 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:02:58.801964 (+ 1934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:02:58.801972 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:02:58.803689 (+ 1717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:02:58.803706 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:02:58.805292 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:02:58.805300 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:02:58.805606 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:02:58.806486 (+ 880us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:02:58.806519 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:02:58.809521 (+ 3002us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:02:58.814628 (+ 5107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:02:58.814648 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:02:58.814678 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:02:58.815541 (+ 863us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:02:58.816494 (+ 953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:02:58.816500 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:02:58.816506 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:02:58.816954 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:02:58.817966 (+ 1012us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:02:58.817977 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:02:58.818435 (+ 458us) client_negotiation.cc:769] Sending connection context
0806 03:02:58.818853 (+ 418us) client_negotiation.cc:240] Negotiation successful
0806 03:02:58.819501 (+ 648us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":346,"threads_started":1}
I20260806 03:02:58.823136 14414 heartbeater.cc:347] Connected to a master server at 127.31.74.254:33865
I20260806 03:02:58.823621 14414 heartbeater.cc:464] Registering TS with master...
I20260806 03:02:58.824757 14414 heartbeater.cc:511] Master 127.31.74.254:33865 requested a full tablet report, sending...
I20260806 03:02:58.827399 13936 ts_manager.cc:194] Registered new tserver with Master: 2ee2abdf415246509365cee48c030d36 (127.31.74.195:32815)
I20260806 03:02:58.828868 13936 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.74.195@KRBTEST.COM'} at 127.31.74.195:33147
I20260806 03:02:58.831262 32043 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 03:02:58.857273 14416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:58.837751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40208 (local address 127.31.74.254:33865)
0806 03:02:58.838108 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0806 03:02:58.838115 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 03:02:58.838572 (+ 457us) server_negotiation.cc:407] Connection header received
0806 03:02:58.838994 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:02:58.838999 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:02:58.839107 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:02:58.839201 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:02:58.841342 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:58.842200 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:58.844437 (+ 2237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:58.845156 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:58.845287 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:02:58.849137 (+ 3850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:02:58.849148 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:02:58.849151 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:02:58.849199 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:02:58.853106 (+ 3907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:02:58.854141 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:02:58.854148 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:02:58.854151 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:02:58.854396 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:02:58.855140 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:02:58.855146 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:02:58.855149 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:02:58.855625 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:02:58.855866 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0806 03:02:58.856737 (+ 871us) server_negotiation.cc:299] Negotiation successful
0806 03:02:58.856979 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166}
I20260806 03:02:58.864670 13936 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40208:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 03:02:58.873319 13936 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 03:02:58.935426 14430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:58.892253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55556 (local address 127.31.74.193:37831)
0806 03:02:58.922069 (+ 29816us) server_negotiation.cc:206] Beginning negotiation
0806 03:02:58.922083 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 03:02:58.922121 (+ 38us) server_negotiation.cc:407] Connection header received
0806 03:02:58.922261 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:02:58.922271 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:02:58.922689 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:02:58.922948 (+ 259us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:02:58.923952 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:58.925943 (+ 1991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:58.930908 (+ 4965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:58.932524 (+ 1616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:58.932773 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:02:58.933063 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0806 03:02:58.933766 (+ 703us) server_negotiation.cc:299] Negotiation successful
0806 03:02:58.933983 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29614,"thread_start_us":29426,"threads_started":1}
I20260806 03:02:58.936477 14433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:58.893520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48466 (local address 127.31.74.194:45731)
0806 03:02:58.916562 (+ 23042us) server_negotiation.cc:206] Beginning negotiation
0806 03:02:58.916581 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 03:02:58.922201 (+ 5620us) server_negotiation.cc:407] Connection header received
0806 03:02:58.922362 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:02:58.922374 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:02:58.922722 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:02:58.922963 (+ 241us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:02:58.924301 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:58.926061 (+ 1760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:58.930867 (+ 4806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:58.933273 (+ 2406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:58.933439 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:02:58.933647 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0806 03:02:58.934421 (+ 774us) server_negotiation.cc:299] Negotiation successful
0806 03:02:58.935429 (+ 1008us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22617,"thread_start_us":23077,"threads_started":1}
I20260806 03:02:58.936590 14431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:58.892427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.195:32815 (local address 127.0.0.1:54688)
0806 03:02:58.921091 (+ 28664us) negotiation.cc:107] Waiting for socket to connect
0806 03:02:58.921116 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 03:02:58.921263 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:02:58.923307 (+ 2044us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:02:58.923311 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:02:58.923362 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:02:58.924425 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:02:58.924438 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:02:58.926314 (+ 1876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:02:58.926317 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:02:58.930374 (+ 4057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:02:58.930394 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:02:58.933874 (+ 3480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:02:58.933880 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:02:58.934126 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:02:58.934234 (+ 108us) client_negotiation.cc:769] Sending connection context
0806 03:02:58.934336 (+ 102us) client_negotiation.cc:240] Negotiation successful
0806 03:02:58.934485 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28500,"mutex_wait_us":772,"spinlock_wait_cycles":7040,"thread_start_us":28252,"threads_started":1}
I20260806 03:02:58.937979 14432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:58.892183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:37831 (local address 127.0.0.1:55556)
0806 03:02:58.918443 (+ 26260us) negotiation.cc:107] Waiting for socket to connect
0806 03:02:58.918524 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0806 03:02:58.918878 (+ 354us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:02:58.922955 (+ 4077us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:02:58.922964 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:02:58.923012 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:02:58.923611 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:02:58.923627 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:02:58.926161 (+ 2534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:02:58.926177 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:02:58.930657 (+ 4480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:02:58.930674 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:02:58.932837 (+ 2163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:02:58.932846 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:02:58.933092 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:02:58.933169 (+ 77us) client_negotiation.cc:769] Sending connection context
0806 03:02:58.933407 (+ 238us) client_negotiation.cc:240] Negotiation successful
0806 03:02:58.934509 (+ 1102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25746,"mutex_wait_us":649,"thread_start_us":29490,"threads_started":1}
I20260806 03:02:58.938171 14428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:58.892489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.194:45731 (local address 127.0.0.1:48466)
0806 03:02:58.922012 (+ 29523us) negotiation.cc:107] Waiting for socket to connect
0806 03:02:58.922035 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 03:02:58.922191 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:02:58.923301 (+ 1110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:02:58.923307 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:02:58.923357 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:02:58.923997 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:02:58.924015 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:02:58.926192 (+ 2177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:02:58.926208 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:02:58.930467 (+ 4259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:02:58.930489 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:02:58.933859 (+ 3370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:02:58.933866 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:02:58.934231 (+ 365us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:02:58.934257 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 03:02:58.937392 (+ 3135us) client_negotiation.cc:240] Negotiation successful
0806 03:02:58.937628 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29382,"mutex_wait_us":344,"spinlock_wait_cycles":1280,"thread_start_us":26857,"threads_started":1}
I20260806 03:02:58.939092 14434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:58.893647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54688 (local address 127.31.74.195:32815)
0806 03:02:58.922612 (+ 28965us) server_negotiation.cc:206] Beginning negotiation
0806 03:02:58.922622 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 03:02:58.922662 (+ 40us) server_negotiation.cc:407] Connection header received
0806 03:02:58.922779 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:02:58.922788 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:02:58.923034 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:02:58.923464 (+ 430us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:02:58.924679 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:58.926168 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:58.930628 (+ 4460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:58.933284 (+ 2656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:58.933440 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:02:58.933709 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0806 03:02:58.938452 (+ 4743us) server_negotiation.cc:299] Negotiation successful
0806 03:02:58.938720 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28733,"thread_start_us":28538,"threads_started":1}
I20260806 03:02:58.943599 14208 tablet_service.cc:1467] Processing CreateTablet for tablet e46517737a014d479db14c4890205bda (DEFAULT_TABLE table=test-table [id=1d907fe541fb4455b39f1d78eadd932b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:02:58.943598 14068 tablet_service.cc:1467] Processing CreateTablet for tablet e46517737a014d479db14c4890205bda (DEFAULT_TABLE table=test-table [id=1d907fe541fb4455b39f1d78eadd932b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:02:58.945370 14068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e46517737a014d479db14c4890205bda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:02:58.945384 14208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e46517737a014d479db14c4890205bda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:02:58.944985 14348 tablet_service.cc:1467] Processing CreateTablet for tablet e46517737a014d479db14c4890205bda (DEFAULT_TABLE table=test-table [id=1d907fe541fb4455b39f1d78eadd932b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:02:58.947086 14348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e46517737a014d479db14c4890205bda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:02:58.966454 14435 tablet_bootstrap.cc:492] T e46517737a014d479db14c4890205bda P 2ee2abdf415246509365cee48c030d36: Bootstrap starting.
I20260806 03:02:58.971608 14436 tablet_bootstrap.cc:492] T e46517737a014d479db14c4890205bda P bb22bd13bfd8404aa9d02498577cd212: Bootstrap starting.
I20260806 03:02:58.971884 14435 tablet_bootstrap.cc:654] T e46517737a014d479db14c4890205bda P 2ee2abdf415246509365cee48c030d36: Neither blocks nor log segments found. Creating new log.
I20260806 03:02:58.974043 14435 log.cc:824] T e46517737a014d479db14c4890205bda P 2ee2abdf415246509365cee48c030d36: Log is configured to *not* fsync() on all Append() calls
I20260806 03:02:58.975996 14437 tablet_bootstrap.cc:492] T e46517737a014d479db14c4890205bda P dc3142ac2c8548c48750d63e7eca5961: Bootstrap starting.
I20260806 03:02:58.977149 14436 tablet_bootstrap.cc:654] T e46517737a014d479db14c4890205bda P bb22bd13bfd8404aa9d02498577cd212: Neither blocks nor log segments found. Creating new log.
I20260806 03:02:58.979212 14436 log.cc:824] T e46517737a014d479db14c4890205bda P bb22bd13bfd8404aa9d02498577cd212: Log is configured to *not* fsync() on all Append() calls
I20260806 03:02:58.980928 14435 tablet_bootstrap.cc:492] T e46517737a014d479db14c4890205bda P 2ee2abdf415246509365cee48c030d36: No bootstrap required, opened a new log
I20260806 03:02:58.981309 14435 ts_tablet_manager.cc:1397] T e46517737a014d479db14c4890205bda P 2ee2abdf415246509365cee48c030d36: Time spent bootstrapping tablet: real 0.018s user 0.010s sys 0.000s
I20260806 03:02:58.981873 14437 tablet_bootstrap.cc:654] T e46517737a014d479db14c4890205bda P dc3142ac2c8548c48750d63e7eca5961: Neither blocks nor log segments found. Creating new log.
I20260806 03:02:58.983080 14437 log.cc:824] T e46517737a014d479db14c4890205bda P dc3142ac2c8548c48750d63e7eca5961: Log is configured to *not* fsync() on all Append() calls
I20260806 03:02:58.983314 14436 tablet_bootstrap.cc:492] T e46517737a014d479db14c4890205bda P bb22bd13bfd8404aa9d02498577cd212: No bootstrap required, opened a new log
I20260806 03:02:58.983634 14436 ts_tablet_manager.cc:1397] T e46517737a014d479db14c4890205bda P bb22bd13bfd8404aa9d02498577cd212: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.002s
I20260806 03:02:58.985976 14437 tablet_bootstrap.cc:492] T e46517737a014d479db14c4890205bda P dc3142ac2c8548c48750d63e7eca5961: No bootstrap required, opened a new log
I20260806 03:02:58.986272 14437 ts_tablet_manager.cc:1397] T e46517737a014d479db14c4890205bda P dc3142ac2c8548c48750d63e7eca5961: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260806 03:02:58.995792 14435 raft_consensus.cc:348] T e46517737a014d479db14c4890205bda P 2ee2abdf415246509365cee48c030d36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc3142ac2c8548c48750d63e7eca5961" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37831 } } peers { permanent_uuid: "2ee2abdf415246509365cee48c030d36" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 32815 } } peers { permanent_uuid: "bb22bd13bfd8404aa9d02498577cd212" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 45731 } }
I20260806 03:02:58.996407 14435 raft_consensus.cc:374] T e46517737a014d479db14c4890205bda P 2ee2abdf415246509365cee48c030d36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:02:58.996604 14435 raft_consensus.cc:729] T e46517737a014d479db14c4890205bda P 2ee2abdf415246509365cee48c030d36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ee2abdf415246509365cee48c030d36, State: Initialized, Role: FOLLOWER
I20260806 03:02:58.997166 14436 raft_consensus.cc:348] T e46517737a014d479db14c4890205bda P bb22bd13bfd8404aa9d02498577cd212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc3142ac2c8548c48750d63e7eca5961" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37831 } } peers { permanent_uuid: "2ee2abdf415246509365cee48c030d36" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 32815 } } peers { permanent_uuid: "bb22bd13bfd8404aa9d02498577cd212" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 45731 } }
I20260806 03:02:58.997680 14436 raft_consensus.cc:374] T e46517737a014d479db14c4890205bda P bb22bd13bfd8404aa9d02498577cd212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:02:58.997550 14435 consensus_queue.cc:260] T e46517737a014d479db14c4890205bda P 2ee2abdf415246509365cee48c030d36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc3142ac2c8548c48750d63e7eca5961" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37831 } } peers { permanent_uuid: "2ee2abdf415246509365cee48c030d36" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 32815 } } peers { permanent_uuid: "bb22bd13bfd8404aa9d02498577cd212" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 45731 } }
I20260806 03:02:58.997820 14436 raft_consensus.cc:729] T e46517737a014d479db14c4890205bda P bb22bd13bfd8404aa9d02498577cd212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb22bd13bfd8404aa9d02498577cd212, State: Initialized, Role: FOLLOWER
I20260806 03:02:58.998274 14437 raft_consensus.cc:348] T e46517737a014d479db14c4890205bda P dc3142ac2c8548c48750d63e7eca5961 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc3142ac2c8548c48750d63e7eca5961" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37831 } } peers { permanent_uuid: "2ee2abdf415246509365cee48c030d36" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 32815 } } peers { permanent_uuid: "bb22bd13bfd8404aa9d02498577cd212" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 45731 } }
I20260806 03:02:58.998687 14437 raft_consensus.cc:374] T e46517737a014d479db14c4890205bda P dc3142ac2c8548c48750d63e7eca5961 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:02:58.998814 14437 raft_consensus.cc:729] T e46517737a014d479db14c4890205bda P dc3142ac2c8548c48750d63e7eca5961 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc3142ac2c8548c48750d63e7eca5961, State: Initialized, Role: FOLLOWER
I20260806 03:02:58.998771 14436 consensus_queue.cc:260] T e46517737a014d479db14c4890205bda P bb22bd13bfd8404aa9d02498577cd212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc3142ac2c8548c48750d63e7eca5961" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37831 } } peers { permanent_uuid: "2ee2abdf415246509365cee48c030d36" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 32815 } } peers { permanent_uuid: "bb22bd13bfd8404aa9d02498577cd212" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 45731 } }
I20260806 03:02:58.999466 14437 consensus_queue.cc:260] T e46517737a014d479db14c4890205bda P dc3142ac2c8548c48750d63e7eca5961 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc3142ac2c8548c48750d63e7eca5961" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37831 } } peers { permanent_uuid: "2ee2abdf415246509365cee48c030d36" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 32815 } } peers { permanent_uuid: "bb22bd13bfd8404aa9d02498577cd212" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 45731 } }
I20260806 03:02:59.000550 14277 heartbeater.cc:503] Master 127.31.74.254:33865 was elected leader, sending a full tablet report...
I20260806 03:02:59.001756 14436 ts_tablet_manager.cc:1428] T e46517737a014d479db14c4890205bda P bb22bd13bfd8404aa9d02498577cd212: Time spent starting tablet: real 0.018s user 0.015s sys 0.001s
I20260806 03:02:59.004365 14437 ts_tablet_manager.cc:1428] T e46517737a014d479db14c4890205bda P dc3142ac2c8548c48750d63e7eca5961: Time spent starting tablet: real 0.018s user 0.011s sys 0.007s
I20260806 03:02:59.005000 14414 heartbeater.cc:503] Master 127.31.74.254:33865 was elected leader, sending a full tablet report...
I20260806 03:02:59.005065 14435 ts_tablet_manager.cc:1428] T e46517737a014d479db14c4890205bda P 2ee2abdf415246509365cee48c030d36: Time spent starting tablet: real 0.024s user 0.017s sys 0.007s
W20260806 03:02:59.040266 14415 tablet.cc:2369] T e46517737a014d479db14c4890205bda P 2ee2abdf415246509365cee48c030d36: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 03:02:59.100112 14278 tablet.cc:2369] T e46517737a014d479db14c4890205bda P bb22bd13bfd8404aa9d02498577cd212: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 03:02:59.186333 14139 tablet.cc:2369] T e46517737a014d479db14c4890205bda P dc3142ac2c8548c48750d63e7eca5961: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 03:02:59.212059 14442 raft_consensus.cc:482] T e46517737a014d479db14c4890205bda P bb22bd13bfd8404aa9d02498577cd212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:02:59.212430 14442 raft_consensus.cc:504] T e46517737a014d479db14c4890205bda P bb22bd13bfd8404aa9d02498577cd212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc3142ac2c8548c48750d63e7eca5961" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37831 } } peers { permanent_uuid: "2ee2abdf415246509365cee48c030d36" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 32815 } } peers { permanent_uuid: "bb22bd13bfd8404aa9d02498577cd212" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 45731 } }
I20260806 03:02:59.214273 14442 leader_election.cc:288] T e46517737a014d479db14c4890205bda P bb22bd13bfd8404aa9d02498577cd212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc3142ac2c8548c48750d63e7eca5961 (127.31.74.193:37831), 2ee2abdf415246509365cee48c030d36 (127.31.74.195:32815)
I20260806 03:02:59.225281 14430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:59.214714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.194:40323 (local address 127.31.74.193:37831)
0806 03:02:59.215013 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0806 03:02:59.215023 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 03:02:59.215461 (+ 438us) server_negotiation.cc:407] Connection header received
0806 03:02:59.215812 (+ 351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:02:59.215817 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:02:59.215920 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:02:59.216015 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:02:59.216727 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:59.218004 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:59.222031 (+ 4027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:59.223708 (+ 1677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:59.223927 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:02:59.224014 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 03:02:59.224750 (+ 736us) server_negotiation.cc:299] Negotiation successful
0806 03:02:59.224928 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260806 03:02:59.225250 14434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:59.215405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.194:56529 (local address 127.31.74.195:32815)
0806 03:02:59.215645 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0806 03:02:59.215652 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 03:02:59.216095 (+ 443us) server_negotiation.cc:407] Connection header received
0806 03:02:59.216246 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:02:59.216250 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:02:59.216309 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:02:59.216382 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:02:59.217057 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:59.218227 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:59.222026 (+ 3799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:59.223678 (+ 1652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:59.223944 (+ 266us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:02:59.224014 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0806 03:02:59.224751 (+ 737us) server_negotiation.cc:299] Negotiation successful
0806 03:02:59.224920 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260806 03:02:59.225392 14445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:59.214569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:37831 (local address 127.31.74.194:40323)
0806 03:02:59.215336 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0806 03:02:59.215353 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 03:02:59.215553 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:02:59.216070 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:02:59.216075 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:02:59.216125 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:02:59.216515 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:02:59.216533 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:02:59.218464 (+ 1931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:02:59.218473 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:02:59.221741 (+ 3268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:02:59.221768 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:02:59.224121 (+ 2353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:02:59.224128 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:02:59.224444 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:02:59.224518 (+ 74us) client_negotiation.cc:769] Sending connection context
0806 03:02:59.224634 (+ 116us) client_negotiation.cc:240] Negotiation successful
0806 03:02:59.224814 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"mutex_wait_us":142,"spinlock_wait_cycles":17664,"thread_start_us":385,"threads_started":1}
I20260806 03:02:59.225417 14444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:59.214627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.195:32815 (local address 127.31.74.194:56529)
0806 03:02:59.215950 (+ 1323us) negotiation.cc:107] Waiting for socket to connect
0806 03:02:59.215967 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 03:02:59.216110 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:02:59.216437 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:02:59.216453 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:02:59.216481 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:02:59.216852 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:02:59.216861 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:02:59.218464 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:02:59.218473 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:02:59.221741 (+ 3268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:02:59.221768 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:02:59.224107 (+ 2339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:02:59.224114 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:02:59.224452 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:02:59.224518 (+ 66us) client_negotiation.cc:769] Sending connection context
0806 03:02:59.224634 (+ 116us) client_negotiation.cc:240] Negotiation successful
0806 03:02:59.224814 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1233,"mutex_wait_us":364,"thread_start_us":999,"threads_started":1}
I20260806 03:02:59.226931 14090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e46517737a014d479db14c4890205bda" candidate_uuid: "bb22bd13bfd8404aa9d02498577cd212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc3142ac2c8548c48750d63e7eca5961" is_pre_election: true
I20260806 03:02:59.226912 14369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e46517737a014d479db14c4890205bda" candidate_uuid: "bb22bd13bfd8404aa9d02498577cd212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ee2abdf415246509365cee48c030d36" is_pre_election: true
I20260806 03:02:59.227609 14090 raft_consensus.cc:2454] T e46517737a014d479db14c4890205bda P dc3142ac2c8548c48750d63e7eca5961 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb22bd13bfd8404aa9d02498577cd212 in term 0.
I20260806 03:02:59.227885 14369 raft_consensus.cc:2454] T e46517737a014d479db14c4890205bda P 2ee2abdf415246509365cee48c030d36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb22bd13bfd8404aa9d02498577cd212 in term 0.
I20260806 03:02:59.228525 14160 leader_election.cc:302] T e46517737a014d479db14c4890205bda P bb22bd13bfd8404aa9d02498577cd212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb22bd13bfd8404aa9d02498577cd212, dc3142ac2c8548c48750d63e7eca5961; no voters:
I20260806 03:02:59.229247 14442 raft_consensus.cc:2781] T e46517737a014d479db14c4890205bda P bb22bd13bfd8404aa9d02498577cd212 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:02:59.229449 14442 raft_consensus.cc:482] T e46517737a014d479db14c4890205bda P bb22bd13bfd8404aa9d02498577cd212 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:02:59.229593 14442 raft_consensus.cc:3037] T e46517737a014d479db14c4890205bda P bb22bd13bfd8404aa9d02498577cd212 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:02:59.232439 14442 raft_consensus.cc:504] T e46517737a014d479db14c4890205bda P bb22bd13bfd8404aa9d02498577cd212 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc3142ac2c8548c48750d63e7eca5961" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37831 } } peers { permanent_uuid: "2ee2abdf415246509365cee48c030d36" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 32815 } } peers { permanent_uuid: "bb22bd13bfd8404aa9d02498577cd212" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 45731 } }
I20260806 03:02:59.233301 14442 leader_election.cc:288] T e46517737a014d479db14c4890205bda P bb22bd13bfd8404aa9d02498577cd212 [CANDIDATE]: Term 1 election: Requested vote from peers dc3142ac2c8548c48750d63e7eca5961 (127.31.74.193:37831), 2ee2abdf415246509365cee48c030d36 (127.31.74.195:32815)
I20260806 03:02:59.233984 14090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e46517737a014d479db14c4890205bda" candidate_uuid: "bb22bd13bfd8404aa9d02498577cd212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc3142ac2c8548c48750d63e7eca5961"
I20260806 03:02:59.234256 14090 raft_consensus.cc:3037] T e46517737a014d479db14c4890205bda P dc3142ac2c8548c48750d63e7eca5961 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:02:59.234251 14369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e46517737a014d479db14c4890205bda" candidate_uuid: "bb22bd13bfd8404aa9d02498577cd212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ee2abdf415246509365cee48c030d36"
I20260806 03:02:59.234699 14369 raft_consensus.cc:3037] T e46517737a014d479db14c4890205bda P 2ee2abdf415246509365cee48c030d36 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:02:59.237066 14090 raft_consensus.cc:2454] T e46517737a014d479db14c4890205bda P dc3142ac2c8548c48750d63e7eca5961 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb22bd13bfd8404aa9d02498577cd212 in term 1.
I20260806 03:02:59.237644 14160 leader_election.cc:302] T e46517737a014d479db14c4890205bda P bb22bd13bfd8404aa9d02498577cd212 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb22bd13bfd8404aa9d02498577cd212, dc3142ac2c8548c48750d63e7eca5961; no voters:
I20260806 03:02:59.238020 14442 raft_consensus.cc:2781] T e46517737a014d479db14c4890205bda P bb22bd13bfd8404aa9d02498577cd212 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:02:59.238128 14369 raft_consensus.cc:2454] T e46517737a014d479db14c4890205bda P 2ee2abdf415246509365cee48c030d36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb22bd13bfd8404aa9d02498577cd212 in term 1.
I20260806 03:02:59.238864 14442 raft_consensus.cc:686] T e46517737a014d479db14c4890205bda P bb22bd13bfd8404aa9d02498577cd212 [term 1 LEADER]: Becoming Leader. State: Replica: bb22bd13bfd8404aa9d02498577cd212, State: Running, Role: LEADER
I20260806 03:02:59.239481 14442 consensus_queue.cc:237] T e46517737a014d479db14c4890205bda P bb22bd13bfd8404aa9d02498577cd212 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc3142ac2c8548c48750d63e7eca5961" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37831 } } peers { permanent_uuid: "2ee2abdf415246509365cee48c030d36" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 32815 } } peers { permanent_uuid: "bb22bd13bfd8404aa9d02498577cd212" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 45731 } }
I20260806 03:02:59.246898 13936 catalog_manager.cc:5495] T e46517737a014d479db14c4890205bda P bb22bd13bfd8404aa9d02498577cd212 reported cstate change: term changed from 0 to 1, leader changed from <none> to bb22bd13bfd8404aa9d02498577cd212 (127.31.74.194). New cstate: current_term: 1 leader_uuid: "bb22bd13bfd8404aa9d02498577cd212" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc3142ac2c8548c48750d63e7eca5961" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ee2abdf415246509365cee48c030d36" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 32815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb22bd13bfd8404aa9d02498577cd212" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 45731 } health_report { overall_health: HEALTHY } } }
I20260806 03:02:59.318209 14433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:59.309782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48470 (local address 127.31.74.194:45731)
0806 03:02:59.309968 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0806 03:02:59.309973 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 03:02:59.309992 (+ 19us) server_negotiation.cc:407] Connection header received
0806 03:02:59.310239 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:02:59.310244 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:02:59.310340 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:02:59.310482 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 03:02:59.311324 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:59.312504 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:59.314429 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:59.314991 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:59.315134 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:02:59.315999 (+ 865us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 03:02:59.317078 (+ 1079us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 03:02:59.317231 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0806 03:02:59.317668 (+ 437us) server_negotiation.cc:299] Negotiation successful
0806 03:02:59.317804 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 03:02:59.329895 14369 raft_consensus.cc:1261] T e46517737a014d479db14c4890205bda P 2ee2abdf415246509365cee48c030d36 [term 1 FOLLOWER]: Refusing update from remote peer bb22bd13bfd8404aa9d02498577cd212: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:02:59.329871 14090 raft_consensus.cc:1261] T e46517737a014d479db14c4890205bda P dc3142ac2c8548c48750d63e7eca5961 [term 1 FOLLOWER]: Refusing update from remote peer bb22bd13bfd8404aa9d02498577cd212: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:02:59.331287 14446 consensus_queue.cc:1035] T e46517737a014d479db14c4890205bda P bb22bd13bfd8404aa9d02498577cd212 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc3142ac2c8548c48750d63e7eca5961" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:02:59.331732 14442 consensus_queue.cc:1035] T e46517737a014d479db14c4890205bda P bb22bd13bfd8404aa9d02498577cd212 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ee2abdf415246509365cee48c030d36" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 32815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:02:59.371193 13936 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40208:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 03:02:59.371592 13936 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40208:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 03:02:59.376276 13936 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f53133eff48348659b64246f2f8317c5: Sending DeleteTablet for 3 replicas of tablet e46517737a014d479db14c4890205bda
I20260806 03:02:59.377650 14068 tablet_service.cc:1514] Processing DeleteTablet for tablet e46517737a014d479db14c4890205bda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:02:59 UTC) from {username='kudu', principal='kudu/127.31.74.254@KRBTEST.COM'} at 127.0.0.1:55556
I20260806 03:02:59.378063 14348 tablet_service.cc:1514] Processing DeleteTablet for tablet e46517737a014d479db14c4890205bda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:02:59 UTC) from {username='kudu', principal='kudu/127.31.74.254@KRBTEST.COM'} at 127.0.0.1:54688
I20260806 03:02:59.378410 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 14000
I20260806 03:02:59.379179 14208 tablet_service.cc:1514] Processing DeleteTablet for tablet e46517737a014d479db14c4890205bda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:02:59 UTC) from {username='kudu', principal='kudu/127.31.74.254@KRBTEST.COM'} at 127.0.0.1:48466
I20260806 03:02:59.388455 14460 tablet_replica.cc:330] T e46517737a014d479db14c4890205bda P dc3142ac2c8548c48750d63e7eca5961: stopping tablet replica
I20260806 03:02:59.389315 14460 raft_consensus.cc:2229] T e46517737a014d479db14c4890205bda P dc3142ac2c8548c48750d63e7eca5961 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:02:59.390123 14460 raft_consensus.cc:2258] T e46517737a014d479db14c4890205bda P dc3142ac2c8548c48750d63e7eca5961 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:02:59.391199 14461 tablet_replica.cc:330] T e46517737a014d479db14c4890205bda P 2ee2abdf415246509365cee48c030d36: stopping tablet replica
I20260806 03:02:59.391597 14461 raft_consensus.cc:2229] T e46517737a014d479db14c4890205bda P 2ee2abdf415246509365cee48c030d36 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:02:59.392041 14461 raft_consensus.cc:2258] T e46517737a014d479db14c4890205bda P 2ee2abdf415246509365cee48c030d36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:02:59.393916 14461 ts_tablet_manager.cc:1905] T e46517737a014d479db14c4890205bda P 2ee2abdf415246509365cee48c030d36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:02:59.394385 14460 ts_tablet_manager.cc:1905] T e46517737a014d479db14c4890205bda P dc3142ac2c8548c48750d63e7eca5961: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:02:59.400873 14462 tablet_replica.cc:330] T e46517737a014d479db14c4890205bda P bb22bd13bfd8404aa9d02498577cd212: stopping tablet replica
I20260806 03:02:59.402029 14462 raft_consensus.cc:2229] T e46517737a014d479db14c4890205bda P bb22bd13bfd8404aa9d02498577cd212 [term 1 LEADER]: Raft consensus shutting down.
I20260806 03:02:59.403162 14462 raft_consensus.cc:2258] T e46517737a014d479db14c4890205bda P bb22bd13bfd8404aa9d02498577cd212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:02:59.403779 14461 ts_tablet_manager.cc:1918] T e46517737a014d479db14c4890205bda P 2ee2abdf415246509365cee48c030d36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 03:02:59.404062 14461 log.cc:1196] T e46517737a014d479db14c4890205bda P 2ee2abdf415246509365cee48c030d36: Deleting WAL directory at /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-2/wal/wals/e46517737a014d479db14c4890205bda
I20260806 03:02:59.404855 14461 ts_tablet_manager.cc:1939] T e46517737a014d479db14c4890205bda P 2ee2abdf415246509365cee48c030d36: Deleting consensus metadata
I20260806 03:02:59.407037 13920 catalog_manager.cc:4841] TS 2ee2abdf415246509365cee48c030d36 (127.31.74.195:32815): tablet e46517737a014d479db14c4890205bda (table test-table [id=1d907fe541fb4455b39f1d78eadd932b]) successfully deleted
I20260806 03:02:59.407661 14462 ts_tablet_manager.cc:1905] T e46517737a014d479db14c4890205bda P bb22bd13bfd8404aa9d02498577cd212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:02:59.409996 14460 ts_tablet_manager.cc:1918] T e46517737a014d479db14c4890205bda P dc3142ac2c8548c48750d63e7eca5961: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 03:02:59.410210 14460 log.cc:1196] T e46517737a014d479db14c4890205bda P dc3142ac2c8548c48750d63e7eca5961: Deleting WAL directory at /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-0/wal/wals/e46517737a014d479db14c4890205bda
I20260806 03:02:59.410868 14460 ts_tablet_manager.cc:1939] T e46517737a014d479db14c4890205bda P dc3142ac2c8548c48750d63e7eca5961: Deleting consensus metadata
I20260806 03:02:59.414224 13919 catalog_manager.cc:4841] TS dc3142ac2c8548c48750d63e7eca5961 (127.31.74.193:37831): tablet e46517737a014d479db14c4890205bda (table test-table [id=1d907fe541fb4455b39f1d78eadd932b]) successfully deleted
I20260806 03:02:59.416015 14462 ts_tablet_manager.cc:1918] T e46517737a014d479db14c4890205bda P bb22bd13bfd8404aa9d02498577cd212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 03:02:59.416285 14462 log.cc:1196] T e46517737a014d479db14c4890205bda P bb22bd13bfd8404aa9d02498577cd212: Deleting WAL directory at /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-1/wal/wals/e46517737a014d479db14c4890205bda
I20260806 03:02:59.417520 14462 ts_tablet_manager.cc:1939] T e46517737a014d479db14c4890205bda P bb22bd13bfd8404aa9d02498577cd212: Deleting consensus metadata
Aug 06 03:02:59 dist-test-slave-dhph krb5kdc[13889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985375, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.74.193@KRBTEST.COM
I20260806 03:02:59.420624 13922 catalog_manager.cc:4841] TS bb22bd13bfd8404aa9d02498577cd212 (127.31.74.194:45731): tablet e46517737a014d479db14c4890205bda (table test-table [id=1d907fe541fb4455b39f1d78eadd932b]) successfully deleted
I20260806 03:02:59.429950 14430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:59.380524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55572 (local address 127.31.74.193:37831)
0806 03:02:59.380744 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0806 03:02:59.380750 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 03:02:59.399408 (+ 18658us) server_negotiation.cc:407] Connection header received
0806 03:02:59.399626 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:02:59.399633 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:02:59.399742 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:02:59.399882 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:02:59.403901 (+ 4019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:59.405162 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:59.407504 (+ 2342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:59.408081 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:59.408165 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:02:59.420911 (+ 12746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:02:59.420939 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:02:59.420959 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:02:59.421019 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:02:59.425523 (+ 4504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:02:59.426505 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:02:59.426517 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:02:59.426521 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:02:59.426769 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:02:59.427525 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:02:59.427530 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:02:59.427534 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:02:59.428087 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:02:59.428496 (+ 409us) server_negotiation.cc:1035] Waiting for connection context
0806 03:02:59.429029 (+ 533us) server_negotiation.cc:299] Negotiation successful
0806 03:02:59.429326 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260806 03:02:59.430895 14132 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 03:02:59.563216 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 14000
I20260806 03:02:59.597236 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 14144
Aug 06 03:02:59 dist-test-slave-dhph krb5kdc[13889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985375, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.74.194@KRBTEST.COM
I20260806 03:02:59.622951 14433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:59.598081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48482 (local address 127.31.74.194:45731)
0806 03:02:59.598273 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0806 03:02:59.598277 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 03:02:59.598318 (+ 41us) server_negotiation.cc:407] Connection header received
0806 03:02:59.598642 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:02:59.598651 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:02:59.598733 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:02:59.598873 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:02:59.601900 (+ 3027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:59.603034 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:59.605485 (+ 2451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:59.606488 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:59.606833 (+ 345us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:02:59.613431 (+ 6598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:02:59.613459 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:02:59.613484 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:02:59.613558 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:02:59.618841 (+ 5283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:02:59.619779 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:02:59.619792 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:02:59.619796 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:02:59.620004 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:02:59.620513 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:02:59.620518 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:02:59.620522 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:02:59.621061 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:02:59.621474 (+ 413us) server_negotiation.cc:1035] Waiting for connection context
0806 03:02:59.622090 (+ 616us) server_negotiation.cc:299] Negotiation successful
0806 03:02:59.622373 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 03:02:59.624047 14271 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 03:02:59.758832 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 14144
I20260806 03:02:59.781950 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 14283
Aug 06 03:02:59 dist-test-slave-dhph krb5kdc[13889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985375, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.74.195@KRBTEST.COM
I20260806 03:02:59.803066 14466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:02:59.783190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54698 (local address 127.31.74.195:32815)
0806 03:02:59.783693 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0806 03:02:59.783698 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 03:02:59.783715 (+ 17us) server_negotiation.cc:407] Connection header received
0806 03:02:59.783771 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:02:59.783774 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:02:59.783843 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:02:59.783960 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:02:59.785340 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:59.786287 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:59.788523 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:02:59.789015 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:02:59.789131 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:02:59.795099 (+ 5968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:02:59.795112 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:02:59.795127 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:02:59.795172 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:02:59.799514 (+ 4342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:02:59.800421 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:02:59.800429 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:02:59.800431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:02:59.800573 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:02:59.801025 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:02:59.801035 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:02:59.801037 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:02:59.801475 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:02:59.801837 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0806 03:02:59.802189 (+ 352us) server_negotiation.cc:299] Negotiation successful
0806 03:02:59.802494 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":278,"threads_started":1}
I20260806 03:02:59.804026 14409 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 03:02:59.915480 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 14283
I20260806 03:02:59.945351 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 13903
I20260806 03:02:59.946938 13968 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 03:03:00.074966 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 13903
I20260806 03:03:00.103739 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:33865
--webserver_interface=127.31.74.254
--webserver_port=39355
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab
--principal=kudu/127.31.74.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:44989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.74.254:33865
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:03:00.526093 14469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:03:00.526489 14469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:03:00.526861 14469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:03:00.536413 14469 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:03:00.536564 14469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:03:00.536684 14469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:03:00.536784 14469 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:03:00.536891 14469 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:03:00.550065 14469 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:44989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.31.74.254:33865
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab
--principal=kudu/127.31.74.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:33865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.74.254
--webserver_port=39355
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:03:00.551673 14469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:03:00.552986 14469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:03:00.584465 14475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:03:00.584501 14477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:03:00.584524 14479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:03:00.585597 14469 server_base.cc:1034] running on GCE node
I20260806 03:03:00.586704 14469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:03:00.588621 14469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:03:00.589929 14469 hybrid_clock.cc:648] HybridClock initialized: now 1785985380589864 us; error 45 us; skew 500 ppm
I20260806 03:03:00.590396 14469 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 03:03:00 dist-test-slave-dhph krb5kdc[13889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985380, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:03:00.594897 14469 init.cc:377] Logged in from keytab as kudu/127.31.74.254@KRBTEST.COM (short username kudu)
I20260806 03:03:00.598227 14469 webserver.cc:460] Webserver started at http://127.31.74.254:39355/ using document root <none> and password file <none>
I20260806 03:03:00.601717 14469 fs_manager.cc:359] Metadata directory not provided
I20260806 03:03:00.602226 14469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:03:00.613999 14469 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.005s sys 0.000s
I20260806 03:03:00.618939 14486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:03:00.620869 14469 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.002s
I20260806 03:03:00.621112 14469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/master-0/wal
uuid: "f53133eff48348659b64246f2f8317c5"
format_stamp: "Formatted at 2026-08-06 03:02:56 on dist-test-slave-dhph"
server_key: "245a2d316f63d43e223222235a5005cc"
server_key_iv: "b2b761d8fb8444f5ee18e31d55c49e36"
server_key_version: "encryptionkey@0"
I20260806 03:03:00.622351 14469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:03:00.651350 14469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:03:00.659895 14469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:03:00.660246 14469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:03:00.688409 14469 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.254:33865
I20260806 03:03:00.688444 14539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.254:33865 every 8 connection(s)
I20260806 03:03:00.690950 14469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
I20260806 03:03:00.695474 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 14469
I20260806 03:03:00.696677 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.193:37831
--local_ip_for_outbound_sockets=127.31.74.193
--tserver_master_addrs=127.31.74.254:33865
--webserver_port=37087
--webserver_interface=127.31.74.193
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.193.keytab
--principal=kudu/127.31.74.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:44989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 03:03:00.700835 14540 sys_catalog.cc:263] Verifying existing consensus state
I20260806 03:03:00.706015 14540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f53133eff48348659b64246f2f8317c5: Bootstrap starting.
I20260806 03:03:00.738886 14540 log.cc:824] T 00000000000000000000000000000000 P f53133eff48348659b64246f2f8317c5: Log is configured to *not* fsync() on all Append() calls
I20260806 03:03:00.755575 14540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f53133eff48348659b64246f2f8317c5: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 03:03:00.756260 14540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f53133eff48348659b64246f2f8317c5: Bootstrap complete.
I20260806 03:03:00.766479 14540 raft_consensus.cc:348] T 00000000000000000000000000000000 P f53133eff48348659b64246f2f8317c5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f53133eff48348659b64246f2f8317c5" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 33865 } }
I20260806 03:03:00.768359 14540 raft_consensus.cc:729] T 00000000000000000000000000000000 P f53133eff48348659b64246f2f8317c5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f53133eff48348659b64246f2f8317c5, State: Initialized, Role: FOLLOWER
I20260806 03:03:00.769395 14540 consensus_queue.cc:260] T 00000000000000000000000000000000 P f53133eff48348659b64246f2f8317c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f53133eff48348659b64246f2f8317c5" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 33865 } }
I20260806 03:03:00.769759 14540 raft_consensus.cc:388] T 00000000000000000000000000000000 P f53133eff48348659b64246f2f8317c5 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:03:00.769919 14540 raft_consensus.cc:482] T 00000000000000000000000000000000 P f53133eff48348659b64246f2f8317c5 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:03:00.770244 14540 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f53133eff48348659b64246f2f8317c5 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:03:00.776815 14540 raft_consensus.cc:504] T 00000000000000000000000000000000 P f53133eff48348659b64246f2f8317c5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f53133eff48348659b64246f2f8317c5" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 33865 } }
I20260806 03:03:00.777467 14540 leader_election.cc:302] T 00000000000000000000000000000000 P f53133eff48348659b64246f2f8317c5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f53133eff48348659b64246f2f8317c5; no voters:
I20260806 03:03:00.778249 14540 leader_election.cc:288] T 00000000000000000000000000000000 P f53133eff48348659b64246f2f8317c5 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 03:03:00.778431 14544 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f53133eff48348659b64246f2f8317c5 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:03:00.779950 14544 raft_consensus.cc:686] T 00000000000000000000000000000000 P f53133eff48348659b64246f2f8317c5 [term 2 LEADER]: Becoming Leader. State: Replica: f53133eff48348659b64246f2f8317c5, State: Running, Role: LEADER
I20260806 03:03:00.780658 14544 consensus_queue.cc:237] T 00000000000000000000000000000000 P f53133eff48348659b64246f2f8317c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f53133eff48348659b64246f2f8317c5" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 33865 } }
I20260806 03:03:00.781220 14540 sys_catalog.cc:564] T 00000000000000000000000000000000 P f53133eff48348659b64246f2f8317c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:03:00.783674 14545 sys_catalog.cc:455] T 00000000000000000000000000000000 P f53133eff48348659b64246f2f8317c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f53133eff48348659b64246f2f8317c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f53133eff48348659b64246f2f8317c5" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 33865 } } }
I20260806 03:03:00.784160 14545 sys_catalog.cc:458] T 00000000000000000000000000000000 P f53133eff48348659b64246f2f8317c5 [sys.catalog]: This master's current role is: LEADER
I20260806 03:03:00.786551 14546 sys_catalog.cc:455] T 00000000000000000000000000000000 P f53133eff48348659b64246f2f8317c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f53133eff48348659b64246f2f8317c5. Latest consensus state: current_term: 2 leader_uuid: "f53133eff48348659b64246f2f8317c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f53133eff48348659b64246f2f8317c5" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 33865 } } }
I20260806 03:03:00.786965 14546 sys_catalog.cc:458] T 00000000000000000000000000000000 P f53133eff48348659b64246f2f8317c5 [sys.catalog]: This master's current role is: LEADER
I20260806 03:03:00.788681 14550 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:03:00.799537 14550 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:03:00.801676 14550 catalog_manager.cc:1259] Loaded cluster ID: 2cd1e9d5e7e44de1b93b1f61ecaf923a
I20260806 03:03:00.802134 14550 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:03:00.806874 14550 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260806 03:03:00.838642 14550 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260806 03:03:00.849465 14550 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:03:00.851783 14550 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f53133eff48348659b64246f2f8317c5: Loaded TSK: 0
I20260806 03:03:00.854759 14550 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 03:03:01.199364 14542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:03:01.199791 14542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:03:01.199990 14542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:03:01.208887 14542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:03:01.209081 14542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:03:01.209672 14542 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.193
I20260806 03:03:01.226186 14542 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:44989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.193.keytab
--principal=kudu/127.31.74.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.193:37831
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.74.193
--webserver_port=37087
--tserver_master_addrs=127.31.74.254:33865
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.193
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:03:01.229058 14542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:03:01.230825 14542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:03:01.252919 14568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:03:01.253750 14571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:03:01.254323 14567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:03:01.256395 14542 server_base.cc:1034] running on GCE node
I20260806 03:03:01.258263 14542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:03:01.260010 14542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:03:01.261458 14542 hybrid_clock.cc:648] HybridClock initialized: now 1785985381261380 us; error 84 us; skew 500 ppm
I20260806 03:03:01.262012 14542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 03:03:01 dist-test-slave-dhph krb5kdc[13889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985381, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:03:01.266918 14542 init.cc:377] Logged in from keytab as kudu/127.31.74.193@KRBTEST.COM (short username kudu)
I20260806 03:03:01.269007 14542 webserver.cc:460] Webserver started at http://127.31.74.193:37087/ using document root <none> and password file <none>
I20260806 03:03:01.269810 14542 fs_manager.cc:359] Metadata directory not provided
I20260806 03:03:01.269984 14542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:03:01.277145 14542 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 03:03:01.281425 14578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:03:01.284484 14542 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260806 03:03:01.284739 14542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-0/wal
uuid: "dc3142ac2c8548c48750d63e7eca5961"
format_stamp: "Formatted at 2026-08-06 03:02:57 on dist-test-slave-dhph"
server_key: "60ac3b964d89f11416c30d1e25045229"
server_key_iv: "8bc41061c44f56028dd17211372facb3"
server_key_version: "encryptionkey@0"
I20260806 03:03:01.285908 14542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:03:01.316185 14542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:03:01.323530 14542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:03:01.323894 14542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:03:01.325194 14542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:03:01.327165 14542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:03:01.327318 14542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:03:01.327488 14542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:03:01.327601 14542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:03:01.380759 14542 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.193:37831
I20260806 03:03:01.380868 14692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.193:37831 every 8 connection(s)
I20260806 03:03:01.382809 14542 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
I20260806 03:03:01.387424 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 14542
I20260806 03:03:01.388675 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.194:45731
--local_ip_for_outbound_sockets=127.31.74.194
--tserver_master_addrs=127.31.74.254:33865
--webserver_port=42495
--webserver_interface=127.31.74.194
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.194.keytab
--principal=kudu/127.31.74.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:44989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 03:03:01 dist-test-slave-dhph krb5kdc[13889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985381, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.193@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 03:03:01.421926 14696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:01.387396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:33865 (local address 127.31.74.193:36241)
0806 03:03:01.389010 (+ 1614us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:01.389067 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0806 03:03:01.391731 (+ 2664us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:03:01.394295 (+ 2564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:03:01.394309 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:03:01.395625 (+ 1316us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:03:01.397038 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:03:01.397089 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:03:01.399295 (+ 2206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:03:01.399300 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:03:01.400847 (+ 1547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:03:01.400865 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:03:01.403216 (+ 2351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:03:01.403226 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:03:01.403533 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:03:01.404343 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:03:01.404371 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:03:01.408358 (+ 3987us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:03:01.414542 (+ 6184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:03:01.414708 (+ 166us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:03:01.414739 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:03:01.415540 (+ 801us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:03:01.416671 (+ 1131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:03:01.416680 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:03:01.416688 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:03:01.417383 (+ 695us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:03:01.419074 (+ 1691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:03:01.419083 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:03:01.419504 (+ 421us) client_negotiation.cc:769] Sending connection context
0806 03:03:01.419817 (+ 313us) client_negotiation.cc:240] Negotiation successful
0806 03:03:01.420424 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1020,"thread_start_us":736,"threads_started":1}
I20260806 03:03:01.422694 14695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:01.386948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.193:36241 (local address 127.31.74.254:33865)
0806 03:03:01.387958 (+ 1010us) server_negotiation.cc:206] Beginning negotiation
0806 03:03:01.388098 (+ 140us) server_negotiation.cc:399] Waiting for connection header
0806 03:03:01.389233 (+ 1135us) server_negotiation.cc:407] Connection header received
0806 03:03:01.392770 (+ 3537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:03:01.392785 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:03:01.393225 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:03:01.394078 (+ 853us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:03:01.397423 (+ 3345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:01.399041 (+ 1618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:01.401486 (+ 2445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:01.402840 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:01.403124 (+ 284us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:03:01.408751 (+ 5627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:03:01.408762 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:03:01.408779 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:03:01.408842 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:03:01.413657 (+ 4815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:03:01.415838 (+ 2181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:03:01.415851 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:03:01.415860 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:03:01.416150 (+ 290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:03:01.417632 (+ 1482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:03:01.417642 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:03:01.417648 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:03:01.418812 (+ 1164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:03:01.419113 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0806 03:03:01.420317 (+ 1204us) server_negotiation.cc:299] Negotiation successful
0806 03:03:01.420997 (+ 680us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":460,"threads_started":1}
I20260806 03:03:01.426615 14693 heartbeater.cc:347] Connected to a master server at 127.31.74.254:33865
I20260806 03:03:01.427054 14693 heartbeater.cc:464] Registering TS with master...
I20260806 03:03:01.428130 14693 heartbeater.cc:511] Master 127.31.74.254:33865 requested a full tablet report, sending...
I20260806 03:03:01.435058 14504 ts_manager.cc:194] Registered new tserver with Master: dc3142ac2c8548c48750d63e7eca5961 (127.31.74.193:37831)
I20260806 03:03:01.438476 14504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.74.193@KRBTEST.COM'} at 127.31.74.193:36241
W20260806 03:03:01.853479 14697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:03:01.853819 14697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:03:01.853921 14697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:03:01.863394 14697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:03:01.863554 14697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:03:01.863684 14697 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.194
I20260806 03:03:01.876667 14697 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:44989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.194.keytab
--principal=kudu/127.31.74.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.194:45731
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.74.194
--webserver_port=42495
--tserver_master_addrs=127.31.74.254:33865
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.194
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:03:01.878157 14697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:03:01.879701 14697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:03:01.900013 14705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:03:01.900108 14702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:03:01.900157 14703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:03:01.901181 14697 server_base.cc:1034] running on GCE node
I20260806 03:03:01.902180 14697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:03:01.904001 14697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:03:01.905328 14697 hybrid_clock.cc:648] HybridClock initialized: now 1785985381905231 us; error 82 us; skew 500 ppm
I20260806 03:03:01.905932 14697 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 03:03:01 dist-test-slave-dhph krb5kdc[13889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985381, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:03:01.910249 14697 init.cc:377] Logged in from keytab as kudu/127.31.74.194@KRBTEST.COM (short username kudu)
I20260806 03:03:01.912396 14697 webserver.cc:460] Webserver started at http://127.31.74.194:42495/ using document root <none> and password file <none>
I20260806 03:03:01.913410 14697 fs_manager.cc:359] Metadata directory not provided
I20260806 03:03:01.913851 14697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:03:01.920786 14697 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 03:03:01.925257 14713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:03:01.928107 14697 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.004s
I20260806 03:03:01.928426 14697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-1/wal
uuid: "bb22bd13bfd8404aa9d02498577cd212"
format_stamp: "Formatted at 2026-08-06 03:02:57 on dist-test-slave-dhph"
server_key: "3afd3b3c8bda75f0f7e6d1aaa9388e51"
server_key_iv: "493b446ae358887437b3a868469f5f6d"
server_key_version: "encryptionkey@0"
I20260806 03:03:01.930317 14697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:03:01.956225 14697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:03:01.962554 14697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:03:01.962941 14697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:03:01.964766 14697 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:03:01.967250 14697 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:03:01.967448 14697 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:03:01.967625 14697 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:03:01.967738 14697 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:03:02.017293 14697 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.194:45731
I20260806 03:03:02.017364 14829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.194:45731 every 8 connection(s)
I20260806 03:03:02.019182 14697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
I20260806 03:03:02.024626 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 14697
I20260806 03:03:02.025274 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.195:32815
--local_ip_for_outbound_sockets=127.31.74.195
--tserver_master_addrs=127.31.74.254:33865
--webserver_port=35447
--webserver_interface=127.31.74.195
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.195.keytab
--principal=kudu/127.31.74.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:44989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 03:03:02 dist-test-slave-dhph krb5kdc[13889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985381, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.194@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 03:03:02.048676 14832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:02.022718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.194:35519 (local address 127.31.74.254:33865)
0806 03:03:02.023276 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0806 03:03:02.023281 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 03:03:02.024414 (+ 1133us) server_negotiation.cc:407] Connection header received
0806 03:03:02.026509 (+ 2095us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:03:02.026515 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:03:02.026849 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:03:02.027021 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:03:02.029081 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:02.029935 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:02.032287 (+ 2352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:02.032765 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:02.032919 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:03:02.040801 (+ 7882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:03:02.040808 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:03:02.040811 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:03:02.040860 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:03:02.044276 (+ 3416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:03:02.045928 (+ 1652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:03:02.045935 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:03:02.045937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:03:02.046107 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:03:02.046671 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:03:02.046680 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:03:02.046683 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:03:02.046983 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:03:02.047158 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0806 03:03:02.047981 (+ 823us) server_negotiation.cc:299] Negotiation successful
0806 03:03:02.048273 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":282,"threads_started":1}
I20260806 03:03:02.049700 14833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:02.023271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:33865 (local address 127.31.74.194:35519)
0806 03:03:02.024150 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:02.024190 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 03:03:02.026181 (+ 1991us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:03:02.027333 (+ 1152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:03:02.027343 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:03:02.028062 (+ 719us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:03:02.028737 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:03:02.028758 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:03:02.030142 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:03:02.030147 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:03:02.032075 (+ 1928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:03:02.032092 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:03:02.033072 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:03:02.033082 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:03:02.033422 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:03:02.034277 (+ 855us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:03:02.034317 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:03:02.040454 (+ 6137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:03:02.044683 (+ 4229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:03:02.044698 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:03:02.044732 (+ 34us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:03:02.045627 (+ 895us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:03:02.046253 (+ 626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:03:02.046260 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:03:02.046265 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:03:02.046513 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:03:02.047131 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:03:02.047140 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:03:02.047556 (+ 416us) client_negotiation.cc:769] Sending connection context
0806 03:03:02.047857 (+ 301us) client_negotiation.cc:240] Negotiation successful
0806 03:03:02.048445 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":378,"threads_started":1}
I20260806 03:03:02.051615 14830 heartbeater.cc:347] Connected to a master server at 127.31.74.254:33865
I20260806 03:03:02.052416 14830 heartbeater.cc:464] Registering TS with master...
I20260806 03:03:02.053843 14830 heartbeater.cc:511] Master 127.31.74.254:33865 requested a full tablet report, sending...
I20260806 03:03:02.055996 14504 ts_manager.cc:194] Registered new tserver with Master: bb22bd13bfd8404aa9d02498577cd212 (127.31.74.194:45731)
I20260806 03:03:02.056875 14504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.74.194@KRBTEST.COM'} at 127.31.74.194:35519
I20260806 03:03:02.443738 14693 heartbeater.cc:503] Master 127.31.74.254:33865 was elected leader, sending a full tablet report...
W20260806 03:03:02.461644 14834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:03:02.461962 14834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:03:02.462128 14834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:03:02.470479 14834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:03:02.470659 14834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:03:02.471007 14834 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.195
I20260806 03:03:02.483176 14834 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:44989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.195.keytab
--principal=kudu/127.31.74.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.195:32815
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.74.195
--webserver_port=35447
--tserver_master_addrs=127.31.74.254:33865
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.195
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:03:02.484692 14834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:03:02.486213 14834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:03:02.503544 14839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:03:02.503561 14842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:03:02.503719 14840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:03:02.506042 14834 server_base.cc:1034] running on GCE node
I20260806 03:03:02.507369 14834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:03:02.509327 14834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:03:02.510746 14834 hybrid_clock.cc:648] HybridClock initialized: now 1785985382510660 us; error 79 us; skew 500 ppm
I20260806 03:03:02.511233 14834 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 03:03:02 dist-test-slave-dhph krb5kdc[13889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985382, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:03:02.515888 14834 init.cc:377] Logged in from keytab as kudu/127.31.74.195@KRBTEST.COM (short username kudu)
I20260806 03:03:02.518010 14834 webserver.cc:460] Webserver started at http://127.31.74.195:35447/ using document root <none> and password file <none>
I20260806 03:03:02.518985 14834 fs_manager.cc:359] Metadata directory not provided
I20260806 03:03:02.519182 14834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:03:02.526072 14834 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.004s
I20260806 03:03:02.529608 14849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:03:02.531715 14834 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 03:03:02.531905 14834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-2/wal
uuid: "2ee2abdf415246509365cee48c030d36"
format_stamp: "Formatted at 2026-08-06 03:02:58 on dist-test-slave-dhph"
server_key: "ec3cee2c35ef448d2ed30c57459f5885"
server_key_iv: "c75b10fff81f46f9977292ab6a2b8c6e"
server_key_version: "encryptionkey@0"
I20260806 03:03:02.533113 14834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:03:02.565240 14834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:03:02.572583 14834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:03:02.572963 14834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:03:02.574295 14834 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:03:02.576069 14834 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:03:02.576184 14834 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:03:02.576344 14834 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:03:02.576438 14834 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:03:02.629539 14834 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.195:32815
I20260806 03:03:02.629642 14966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.195:32815 every 8 connection(s)
I20260806 03:03:02.631873 14834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
I20260806 03:03:02.640929 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 14834
Aug 06 03:03:02 dist-test-slave-dhph krb5kdc[13889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985382, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.195@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 03:03:02.663300 14969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:02.635811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.195:34883 (local address 127.31.74.254:33865)
0806 03:03:02.636424 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0806 03:03:02.636430 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 03:03:02.638748 (+ 2318us) server_negotiation.cc:407] Connection header received
0806 03:03:02.641117 (+ 2369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:03:02.641123 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:03:02.641263 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:03:02.641447 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:03:02.644532 (+ 3085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:02.646030 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:02.648882 (+ 2852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:02.649326 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:02.649447 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:03:02.655211 (+ 5764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:03:02.655220 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:03:02.655227 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:03:02.655279 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:03:02.658952 (+ 3673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:03:02.659990 (+ 1038us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:03:02.659997 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:03:02.660001 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:03:02.660203 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:03:02.660788 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:03:02.660792 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:03:02.660795 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:03:02.661141 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:03:02.661354 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0806 03:03:02.662444 (+ 1090us) server_negotiation.cc:299] Negotiation successful
0806 03:03:02.662812 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":320,"threads_started":1}
I20260806 03:03:02.664762 14970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:02.636508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:33865 (local address 127.31.74.195:34883)
0806 03:03:02.638473 (+ 1965us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:02.638534 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0806 03:03:02.640507 (+ 1973us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:03:02.641745 (+ 1238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:03:02.641754 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:03:02.642480 (+ 726us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:03:02.643864 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:03:02.643892 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:03:02.646541 (+ 2649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:03:02.646547 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:03:02.648635 (+ 2088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:03:02.648661 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:03:02.649630 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:03:02.649638 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:03:02.649874 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:03:02.650723 (+ 849us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:03:02.650777 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:03:02.654650 (+ 3873us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:03:02.659216 (+ 4566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:03:02.659227 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:03:02.659245 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:03:02.659808 (+ 563us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:03:02.660355 (+ 547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:03:02.660361 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:03:02.660364 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:03:02.660608 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:03:02.661324 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:03:02.661334 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:03:02.661883 (+ 549us) client_negotiation.cc:769] Sending connection context
0806 03:03:02.662268 (+ 385us) client_negotiation.cc:240] Negotiation successful
0806 03:03:02.662996 (+ 728us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1570,"thread_start_us":1310,"threads_started":1}
I20260806 03:03:02.665745 14972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:02.642432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40214 (local address 127.31.74.254:33865)
0806 03:03:02.647130 (+ 4698us) server_negotiation.cc:206] Beginning negotiation
0806 03:03:02.647135 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 03:03:02.647152 (+ 17us) server_negotiation.cc:407] Connection header received
0806 03:03:02.647200 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:03:02.647204 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:03:02.647287 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:03:02.647397 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:03:02.649721 (+ 2324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:02.650681 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:02.652944 (+ 2263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:02.653881 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:02.654005 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:03:02.657702 (+ 3697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:03:02.657714 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:03:02.657720 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:03:02.657792 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:03:02.661412 (+ 3620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:03:02.662487 (+ 1075us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:03:02.662493 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:03:02.662496 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:03:02.662688 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:03:02.663225 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:03:02.663230 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:03:02.663234 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:03:02.663681 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:03:02.663903 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0806 03:03:02.664447 (+ 544us) server_negotiation.cc:299] Negotiation successful
0806 03:03:02.664709 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4581,"thread_start_us":4440,"threads_started":1}
I20260806 03:03:02.667160 14967 heartbeater.cc:347] Connected to a master server at 127.31.74.254:33865
I20260806 03:03:02.667593 14967 heartbeater.cc:464] Registering TS with master...
I20260806 03:03:02.668957 14967 heartbeater.cc:511] Master 127.31.74.254:33865 requested a full tablet report, sending...
I20260806 03:03:02.671051 14504 ts_manager.cc:194] Registered new tserver with Master: 2ee2abdf415246509365cee48c030d36 (127.31.74.195:32815)
I20260806 03:03:02.672092 14504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.74.195@KRBTEST.COM'} at 127.31.74.195:34883
I20260806 03:03:02.684013 32043 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 03:03:02.694516 14972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:02.685334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40220 (local address 127.31.74.254:33865)
0806 03:03:02.685578 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0806 03:03:02.685584 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 03:03:02.686162 (+ 578us) server_negotiation.cc:407] Connection header received
0806 03:03:02.686466 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:03:02.686470 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:03:02.686792 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:03:02.686950 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 03:03:02.687922 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:02.688816 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:02.691053 (+ 2237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:02.691871 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:02.692035 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:03:02.692913 (+ 878us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 03:03:02.693690 (+ 777us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 03:03:02.693864 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0806 03:03:02.694049 (+ 185us) server_negotiation.cc:299] Negotiation successful
0806 03:03:02.694155 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260806 03:03:02.695833 14504 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:40220:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 03:03:02.705626 14504 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 03:03:02.750711 14982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:02.722375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48498 (local address 127.31.74.194:45731)
0806 03:03:02.740414 (+ 18039us) server_negotiation.cc:206] Beginning negotiation
0806 03:03:02.740429 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 03:03:02.741566 (+ 1137us) server_negotiation.cc:407] Connection header received
0806 03:03:02.741703 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:03:02.741937 (+ 234us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:03:02.742289 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:03:02.742569 (+ 280us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:03:02.743638 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:02.745133 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:02.747949 (+ 2816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:02.749007 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:02.749177 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:03:02.749414 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0806 03:03:02.750162 (+ 748us) server_negotiation.cc:299] Negotiation successful
0806 03:03:02.750357 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17769,"thread_start_us":18065,"threads_started":1}
I20260806 03:03:02.750711 14979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:02.721931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.194:45731 (local address 127.0.0.1:48498)
0806 03:03:02.741190 (+ 19259us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:02.741225 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0806 03:03:02.741396 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:03:02.742887 (+ 1491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:03:02.742891 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:03:02.742941 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:03:02.743268 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:03:02.743281 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:03:02.745323 (+ 2042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:03:02.745331 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:03:02.747752 (+ 2421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:03:02.747767 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:03:02.749374 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:03:02.749381 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:03:02.749596 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:03:02.749651 (+ 55us) client_negotiation.cc:769] Sending connection context
0806 03:03:02.749797 (+ 146us) client_negotiation.cc:240] Negotiation successful
0806 03:03:02.750274 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19088,"thread_start_us":16489,"threads_started":1}
I20260806 03:03:02.754501 14981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:02.722323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55588 (local address 127.31.74.193:37831)
0806 03:03:02.743066 (+ 20743us) server_negotiation.cc:206] Beginning negotiation
0806 03:03:02.743077 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 03:03:02.743119 (+ 42us) server_negotiation.cc:407] Connection header received
0806 03:03:02.743253 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:03:02.743264 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:03:02.743827 (+ 563us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:03:02.744084 (+ 257us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:03:02.745256 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:02.746918 (+ 1662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:02.750122 (+ 3204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:02.752874 (+ 2752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:02.753068 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:03:02.753349 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0806 03:03:02.753911 (+ 562us) server_negotiation.cc:299] Negotiation successful
0806 03:03:02.754131 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20427,"thread_start_us":20180,"threads_started":1}
I20260806 03:03:02.755048 14980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:02.721886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54706 (local address 127.31.74.195:32815)
0806 03:03:02.734923 (+ 13037us) server_negotiation.cc:206] Beginning negotiation
0806 03:03:02.734954 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0806 03:03:02.739117 (+ 4163us) server_negotiation.cc:407] Connection header received
0806 03:03:02.740721 (+ 1604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:03:02.740732 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:03:02.741012 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:03:02.741264 (+ 252us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:03:02.742883 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:02.745922 (+ 3039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:02.750727 (+ 4805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:02.752906 (+ 2179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:02.753068 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:03:02.753346 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0806 03:03:02.754190 (+ 844us) server_negotiation.cc:299] Negotiation successful
0806 03:03:02.754518 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12542,"thread_start_us":12704,"threads_started":1}
I20260806 03:03:02.755403 14978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:02.721575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:37831 (local address 127.0.0.1:55588)
0806 03:03:02.734518 (+ 12943us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:02.734879 (+ 361us) client_negotiation.cc:174] Beginning negotiation
0806 03:03:02.735196 (+ 317us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:03:02.744093 (+ 8897us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:03:02.744098 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:03:02.744128 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:03:02.744803 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:03:02.744818 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:03:02.747158 (+ 2340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:03:02.747168 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:03:02.749885 (+ 2717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:03:02.749902 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:03:02.753389 (+ 3487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:03:02.753399 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:03:02.753616 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:03:02.753663 (+ 47us) client_negotiation.cc:769] Sending connection context
0806 03:03:02.753807 (+ 144us) client_negotiation.cc:240] Negotiation successful
0806 03:03:02.753972 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12531,"mutex_wait_us":1,"thread_start_us":12296,"threads_started":1}
I20260806 03:03:02.754547 14983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:02.721575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.195:32815 (local address 127.0.0.1:54706)
0806 03:03:02.738913 (+ 17338us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:02.738945 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0806 03:03:02.739170 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:03:02.741279 (+ 2109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:03:02.741287 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:03:02.741351 (+ 64us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:03:02.742315 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:03:02.742359 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:03:02.746251 (+ 3892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:03:02.746260 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:03:02.750228 (+ 3968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:03:02.750242 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:03:02.753552 (+ 3310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:03:02.753558 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:03:02.753692 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:03:02.753717 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 03:03:02.753849 (+ 132us) client_negotiation.cc:240] Negotiation successful
0806 03:03:02.754008 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17052,"mutex_wait_us":310,"thread_start_us":19690,"threads_started":1}
I20260806 03:03:02.756783 14763 tablet_service.cc:1467] Processing CreateTablet for tablet dbeeb9af3ace42c68c40c02a7ba33c3d (DEFAULT_TABLE table=test-table [id=7348ff3e3b54480cba9f8af688e83e62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:03:02.758912 14763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbeeb9af3ace42c68c40c02a7ba33c3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:03:02.759824 14627 tablet_service.cc:1467] Processing CreateTablet for tablet dbeeb9af3ace42c68c40c02a7ba33c3d (DEFAULT_TABLE table=test-table [id=7348ff3e3b54480cba9f8af688e83e62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:03:02.759891 14899 tablet_service.cc:1467] Processing CreateTablet for tablet dbeeb9af3ace42c68c40c02a7ba33c3d (DEFAULT_TABLE table=test-table [id=7348ff3e3b54480cba9f8af688e83e62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:03:02.761137 14627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbeeb9af3ace42c68c40c02a7ba33c3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:03:02.761271 14899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbeeb9af3ace42c68c40c02a7ba33c3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:03:02.775421 14984 tablet_bootstrap.cc:492] T dbeeb9af3ace42c68c40c02a7ba33c3d P dc3142ac2c8548c48750d63e7eca5961: Bootstrap starting.
I20260806 03:03:02.777751 14986 tablet_bootstrap.cc:492] T dbeeb9af3ace42c68c40c02a7ba33c3d P 2ee2abdf415246509365cee48c030d36: Bootstrap starting.
I20260806 03:03:02.779587 14985 tablet_bootstrap.cc:492] T dbeeb9af3ace42c68c40c02a7ba33c3d P bb22bd13bfd8404aa9d02498577cd212: Bootstrap starting.
I20260806 03:03:02.780272 14984 tablet_bootstrap.cc:654] T dbeeb9af3ace42c68c40c02a7ba33c3d P dc3142ac2c8548c48750d63e7eca5961: Neither blocks nor log segments found. Creating new log.
I20260806 03:03:02.781950 14986 tablet_bootstrap.cc:654] T dbeeb9af3ace42c68c40c02a7ba33c3d P 2ee2abdf415246509365cee48c030d36: Neither blocks nor log segments found. Creating new log.
I20260806 03:03:02.782735 14985 tablet_bootstrap.cc:654] T dbeeb9af3ace42c68c40c02a7ba33c3d P bb22bd13bfd8404aa9d02498577cd212: Neither blocks nor log segments found. Creating new log.
I20260806 03:03:02.781919 14984 log.cc:824] T dbeeb9af3ace42c68c40c02a7ba33c3d P dc3142ac2c8548c48750d63e7eca5961: Log is configured to *not* fsync() on all Append() calls
I20260806 03:03:02.783958 14985 log.cc:824] T dbeeb9af3ace42c68c40c02a7ba33c3d P bb22bd13bfd8404aa9d02498577cd212: Log is configured to *not* fsync() on all Append() calls
I20260806 03:03:02.784407 14986 log.cc:824] T dbeeb9af3ace42c68c40c02a7ba33c3d P 2ee2abdf415246509365cee48c030d36: Log is configured to *not* fsync() on all Append() calls
I20260806 03:03:02.787432 14984 tablet_bootstrap.cc:492] T dbeeb9af3ace42c68c40c02a7ba33c3d P dc3142ac2c8548c48750d63e7eca5961: No bootstrap required, opened a new log
I20260806 03:03:02.787554 14985 tablet_bootstrap.cc:492] T dbeeb9af3ace42c68c40c02a7ba33c3d P bb22bd13bfd8404aa9d02498577cd212: No bootstrap required, opened a new log
I20260806 03:03:02.787724 14986 tablet_bootstrap.cc:492] T dbeeb9af3ace42c68c40c02a7ba33c3d P 2ee2abdf415246509365cee48c030d36: No bootstrap required, opened a new log
I20260806 03:03:02.787777 14984 ts_tablet_manager.cc:1397] T dbeeb9af3ace42c68c40c02a7ba33c3d P dc3142ac2c8548c48750d63e7eca5961: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.004s
I20260806 03:03:02.787846 14985 ts_tablet_manager.cc:1397] T dbeeb9af3ace42c68c40c02a7ba33c3d P bb22bd13bfd8404aa9d02498577cd212: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260806 03:03:02.787997 14986 ts_tablet_manager.cc:1397] T dbeeb9af3ace42c68c40c02a7ba33c3d P 2ee2abdf415246509365cee48c030d36: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260806 03:03:02.797881 14984 raft_consensus.cc:348] T dbeeb9af3ace42c68c40c02a7ba33c3d P dc3142ac2c8548c48750d63e7eca5961 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ee2abdf415246509365cee48c030d36" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 32815 } } peers { permanent_uuid: "dc3142ac2c8548c48750d63e7eca5961" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37831 } } peers { permanent_uuid: "bb22bd13bfd8404aa9d02498577cd212" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 45731 } }
I20260806 03:03:02.798420 14984 raft_consensus.cc:374] T dbeeb9af3ace42c68c40c02a7ba33c3d P dc3142ac2c8548c48750d63e7eca5961 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:03:02.798537 14984 raft_consensus.cc:729] T dbeeb9af3ace42c68c40c02a7ba33c3d P dc3142ac2c8548c48750d63e7eca5961 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc3142ac2c8548c48750d63e7eca5961, State: Initialized, Role: FOLLOWER
I20260806 03:03:02.799332 14984 consensus_queue.cc:260] T dbeeb9af3ace42c68c40c02a7ba33c3d P dc3142ac2c8548c48750d63e7eca5961 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ee2abdf415246509365cee48c030d36" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 32815 } } peers { permanent_uuid: "dc3142ac2c8548c48750d63e7eca5961" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37831 } } peers { permanent_uuid: "bb22bd13bfd8404aa9d02498577cd212" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 45731 } }
I20260806 03:03:02.799707 14986 raft_consensus.cc:348] T dbeeb9af3ace42c68c40c02a7ba33c3d P 2ee2abdf415246509365cee48c030d36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ee2abdf415246509365cee48c030d36" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 32815 } } peers { permanent_uuid: "dc3142ac2c8548c48750d63e7eca5961" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37831 } } peers { permanent_uuid: "bb22bd13bfd8404aa9d02498577cd212" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 45731 } }
I20260806 03:03:02.799702 14985 raft_consensus.cc:348] T dbeeb9af3ace42c68c40c02a7ba33c3d P bb22bd13bfd8404aa9d02498577cd212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ee2abdf415246509365cee48c030d36" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 32815 } } peers { permanent_uuid: "dc3142ac2c8548c48750d63e7eca5961" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37831 } } peers { permanent_uuid: "bb22bd13bfd8404aa9d02498577cd212" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 45731 } }
I20260806 03:03:02.800199 14986 raft_consensus.cc:374] T dbeeb9af3ace42c68c40c02a7ba33c3d P 2ee2abdf415246509365cee48c030d36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:03:02.800209 14985 raft_consensus.cc:374] T dbeeb9af3ace42c68c40c02a7ba33c3d P bb22bd13bfd8404aa9d02498577cd212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:03:02.800310 14986 raft_consensus.cc:729] T dbeeb9af3ace42c68c40c02a7ba33c3d P 2ee2abdf415246509365cee48c030d36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ee2abdf415246509365cee48c030d36, State: Initialized, Role: FOLLOWER
I20260806 03:03:02.800310 14985 raft_consensus.cc:729] T dbeeb9af3ace42c68c40c02a7ba33c3d P bb22bd13bfd8404aa9d02498577cd212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb22bd13bfd8404aa9d02498577cd212, State: Initialized, Role: FOLLOWER
I20260806 03:03:02.800959 14986 consensus_queue.cc:260] T dbeeb9af3ace42c68c40c02a7ba33c3d P 2ee2abdf415246509365cee48c030d36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ee2abdf415246509365cee48c030d36" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 32815 } } peers { permanent_uuid: "dc3142ac2c8548c48750d63e7eca5961" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37831 } } peers { permanent_uuid: "bb22bd13bfd8404aa9d02498577cd212" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 45731 } }
I20260806 03:03:02.800959 14985 consensus_queue.cc:260] T dbeeb9af3ace42c68c40c02a7ba33c3d P bb22bd13bfd8404aa9d02498577cd212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ee2abdf415246509365cee48c030d36" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 32815 } } peers { permanent_uuid: "dc3142ac2c8548c48750d63e7eca5961" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37831 } } peers { permanent_uuid: "bb22bd13bfd8404aa9d02498577cd212" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 45731 } }
I20260806 03:03:02.802445 14984 ts_tablet_manager.cc:1428] T dbeeb9af3ace42c68c40c02a7ba33c3d P dc3142ac2c8548c48750d63e7eca5961: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
I20260806 03:03:02.804445 14967 heartbeater.cc:503] Master 127.31.74.254:33865 was elected leader, sending a full tablet report...
I20260806 03:03:02.804584 14830 heartbeater.cc:503] Master 127.31.74.254:33865 was elected leader, sending a full tablet report...
I20260806 03:03:02.804926 14986 ts_tablet_manager.cc:1428] T dbeeb9af3ace42c68c40c02a7ba33c3d P 2ee2abdf415246509365cee48c030d36: Time spent starting tablet: real 0.017s user 0.013s sys 0.003s
I20260806 03:03:02.804944 14985 ts_tablet_manager.cc:1428] T dbeeb9af3ace42c68c40c02a7ba33c3d P bb22bd13bfd8404aa9d02498577cd212: Time spent starting tablet: real 0.017s user 0.017s sys 0.001s
I20260806 03:03:02.865911 14990 raft_consensus.cc:482] T dbeeb9af3ace42c68c40c02a7ba33c3d P dc3142ac2c8548c48750d63e7eca5961 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:03:02.866263 14990 raft_consensus.cc:504] T dbeeb9af3ace42c68c40c02a7ba33c3d P dc3142ac2c8548c48750d63e7eca5961 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ee2abdf415246509365cee48c030d36" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 32815 } } peers { permanent_uuid: "dc3142ac2c8548c48750d63e7eca5961" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37831 } } peers { permanent_uuid: "bb22bd13bfd8404aa9d02498577cd212" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 45731 } }
I20260806 03:03:02.868422 14990 leader_election.cc:288] T dbeeb9af3ace42c68c40c02a7ba33c3d P dc3142ac2c8548c48750d63e7eca5961 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ee2abdf415246509365cee48c030d36 (127.31.74.195:32815), bb22bd13bfd8404aa9d02498577cd212 (127.31.74.194:45731)
I20260806 03:03:02.879421 14980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:02.869312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.193:34659 (local address 127.31.74.195:32815)
0806 03:03:02.869562 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0806 03:03:02.869571 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 03:03:02.869846 (+ 275us) server_negotiation.cc:407] Connection header received
0806 03:03:02.870224 (+ 378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:03:02.870230 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:03:02.870335 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:03:02.870441 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:03:02.871403 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:02.872717 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:02.876642 (+ 3925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:02.877977 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:02.878177 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:03:02.878255 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 03:03:02.878935 (+ 680us) server_negotiation.cc:299] Negotiation successful
0806 03:03:02.879096 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260806 03:03:02.879423 14982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:02.869099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.193:58259 (local address 127.31.74.194:45731)
0806 03:03:02.869369 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0806 03:03:02.869377 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 03:03:02.870099 (+ 722us) server_negotiation.cc:407] Connection header received
0806 03:03:02.870390 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:03:02.870398 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:03:02.870488 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:03:02.870609 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:03:02.871487 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:02.872708 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:02.876649 (+ 3941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:02.877977 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:02.878173 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:03:02.878255 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0806 03:03:02.878935 (+ 680us) server_negotiation.cc:299] Negotiation successful
0806 03:03:02.879096 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260806 03:03:02.879701 14995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:02.869074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.194:45731 (local address 127.31.74.193:58259)
0806 03:03:02.869955 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:02.869976 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 03:03:02.870136 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:03:02.870693 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:03:02.870698 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:03:02.870747 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:03:02.871310 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:03:02.871330 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:03:02.873015 (+ 1685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:03:02.873024 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:03:02.876375 (+ 3351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:03:02.876398 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:03:02.878333 (+ 1935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:03:02.878352 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:03:02.878685 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:03:02.878742 (+ 57us) client_negotiation.cc:769] Sending connection context
0806 03:03:02.878862 (+ 120us) client_negotiation.cc:240] Negotiation successful
0806 03:03:02.879115 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":735,"mutex_wait_us":422,"thread_start_us":592,"threads_started":1}
I20260806 03:03:02.880074 14994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:02.868742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.195:32815 (local address 127.31.74.193:34659)
0806 03:03:02.869674 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:02.869710 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0806 03:03:02.869996 (+ 286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:03:02.870489 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:03:02.870496 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:03:02.870528 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:03:02.871229 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:03:02.871250 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:03:02.873015 (+ 1765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:03:02.873024 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:03:02.876376 (+ 3352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:03:02.876398 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:03:02.878373 (+ 1975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:03:02.878379 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:03:02.878675 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:03:02.878737 (+ 62us) client_negotiation.cc:769] Sending connection context
0806 03:03:02.878862 (+ 125us) client_negotiation.cc:240] Negotiation successful
0806 03:03:02.879115 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":701,"mutex_wait_us":278,"thread_start_us":808,"threads_started":1}
I20260806 03:03:02.881079 14783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbeeb9af3ace42c68c40c02a7ba33c3d" candidate_uuid: "dc3142ac2c8548c48750d63e7eca5961" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb22bd13bfd8404aa9d02498577cd212" is_pre_election: true
I20260806 03:03:02.881319 14919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbeeb9af3ace42c68c40c02a7ba33c3d" candidate_uuid: "dc3142ac2c8548c48750d63e7eca5961" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ee2abdf415246509365cee48c030d36" is_pre_election: true
I20260806 03:03:02.881691 14783 raft_consensus.cc:2454] T dbeeb9af3ace42c68c40c02a7ba33c3d P bb22bd13bfd8404aa9d02498577cd212 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc3142ac2c8548c48750d63e7eca5961 in term 0.
I20260806 03:03:02.881892 14919 raft_consensus.cc:2454] T dbeeb9af3ace42c68c40c02a7ba33c3d P 2ee2abdf415246509365cee48c030d36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc3142ac2c8548c48750d63e7eca5961 in term 0.
I20260806 03:03:02.882896 14582 leader_election.cc:302] T dbeeb9af3ace42c68c40c02a7ba33c3d P dc3142ac2c8548c48750d63e7eca5961 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb22bd13bfd8404aa9d02498577cd212, dc3142ac2c8548c48750d63e7eca5961; no voters:
I20260806 03:03:02.883628 14990 raft_consensus.cc:2781] T dbeeb9af3ace42c68c40c02a7ba33c3d P dc3142ac2c8548c48750d63e7eca5961 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:03:02.883791 14990 raft_consensus.cc:482] T dbeeb9af3ace42c68c40c02a7ba33c3d P dc3142ac2c8548c48750d63e7eca5961 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:03:02.883916 14990 raft_consensus.cc:3037] T dbeeb9af3ace42c68c40c02a7ba33c3d P dc3142ac2c8548c48750d63e7eca5961 [term 0 FOLLOWER]: Advancing to term 1
W20260806 03:03:02.884122 14968 tablet.cc:2369] T dbeeb9af3ace42c68c40c02a7ba33c3d P 2ee2abdf415246509365cee48c030d36: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 03:03:02.886150 14694 tablet.cc:2369] T dbeeb9af3ace42c68c40c02a7ba33c3d P dc3142ac2c8548c48750d63e7eca5961: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 03:03:02.887184 14990 raft_consensus.cc:504] T dbeeb9af3ace42c68c40c02a7ba33c3d P dc3142ac2c8548c48750d63e7eca5961 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ee2abdf415246509365cee48c030d36" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 32815 } } peers { permanent_uuid: "dc3142ac2c8548c48750d63e7eca5961" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37831 } } peers { permanent_uuid: "bb22bd13bfd8404aa9d02498577cd212" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 45731 } }
I20260806 03:03:02.887943 14990 leader_election.cc:288] T dbeeb9af3ace42c68c40c02a7ba33c3d P dc3142ac2c8548c48750d63e7eca5961 [CANDIDATE]: Term 1 election: Requested vote from peers 2ee2abdf415246509365cee48c030d36 (127.31.74.195:32815), bb22bd13bfd8404aa9d02498577cd212 (127.31.74.194:45731)
I20260806 03:03:02.888610 14919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbeeb9af3ace42c68c40c02a7ba33c3d" candidate_uuid: "dc3142ac2c8548c48750d63e7eca5961" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ee2abdf415246509365cee48c030d36"
I20260806 03:03:02.888712 14783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbeeb9af3ace42c68c40c02a7ba33c3d" candidate_uuid: "dc3142ac2c8548c48750d63e7eca5961" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb22bd13bfd8404aa9d02498577cd212"
I20260806 03:03:02.888861 14919 raft_consensus.cc:3037] T dbeeb9af3ace42c68c40c02a7ba33c3d P 2ee2abdf415246509365cee48c030d36 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:03:02.889024 14783 raft_consensus.cc:3037] T dbeeb9af3ace42c68c40c02a7ba33c3d P bb22bd13bfd8404aa9d02498577cd212 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:03:02.892015 14783 raft_consensus.cc:2454] T dbeeb9af3ace42c68c40c02a7ba33c3d P bb22bd13bfd8404aa9d02498577cd212 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc3142ac2c8548c48750d63e7eca5961 in term 1.
I20260806 03:03:02.892658 14919 raft_consensus.cc:2454] T dbeeb9af3ace42c68c40c02a7ba33c3d P 2ee2abdf415246509365cee48c030d36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc3142ac2c8548c48750d63e7eca5961 in term 1.
I20260806 03:03:02.892740 14582 leader_election.cc:302] T dbeeb9af3ace42c68c40c02a7ba33c3d P dc3142ac2c8548c48750d63e7eca5961 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb22bd13bfd8404aa9d02498577cd212, dc3142ac2c8548c48750d63e7eca5961; no voters:
I20260806 03:03:02.893213 14990 raft_consensus.cc:2781] T dbeeb9af3ace42c68c40c02a7ba33c3d P dc3142ac2c8548c48750d63e7eca5961 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:03:02.893810 14990 raft_consensus.cc:686] T dbeeb9af3ace42c68c40c02a7ba33c3d P dc3142ac2c8548c48750d63e7eca5961 [term 1 LEADER]: Becoming Leader. State: Replica: dc3142ac2c8548c48750d63e7eca5961, State: Running, Role: LEADER
I20260806 03:03:02.894382 14990 consensus_queue.cc:237] T dbeeb9af3ace42c68c40c02a7ba33c3d P dc3142ac2c8548c48750d63e7eca5961 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ee2abdf415246509365cee48c030d36" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 32815 } } peers { permanent_uuid: "dc3142ac2c8548c48750d63e7eca5961" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37831 } } peers { permanent_uuid: "bb22bd13bfd8404aa9d02498577cd212" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 45731 } }
I20260806 03:03:02.899921 14502 catalog_manager.cc:5495] T dbeeb9af3ace42c68c40c02a7ba33c3d P dc3142ac2c8548c48750d63e7eca5961 reported cstate change: term changed from 0 to 1, leader changed from <none> to dc3142ac2c8548c48750d63e7eca5961 (127.31.74.193). New cstate: current_term: 1 leader_uuid: "dc3142ac2c8548c48750d63e7eca5961" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ee2abdf415246509365cee48c030d36" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 32815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc3142ac2c8548c48750d63e7eca5961" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb22bd13bfd8404aa9d02498577cd212" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 45731 } health_report { overall_health: UNKNOWN } } }
I20260806 03:03:02.959239 14981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:02.951817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55600 (local address 127.31.74.193:37831)
0806 03:03:02.952067 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0806 03:03:02.952073 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 03:03:02.952096 (+ 23us) server_negotiation.cc:407] Connection header received
0806 03:03:02.952233 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:03:02.952238 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:03:02.952355 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:03:02.952593 (+ 238us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 03:03:02.953165 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:02.954102 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:02.956307 (+ 2205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:02.956874 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:02.957117 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:03:02.957768 (+ 651us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 03:03:02.958505 (+ 737us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 03:03:02.958710 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0806 03:03:02.958886 (+ 176us) server_negotiation.cc:299] Negotiation successful
0806 03:03:02.958998 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 03:03:02.969982 14919 raft_consensus.cc:1261] T dbeeb9af3ace42c68c40c02a7ba33c3d P 2ee2abdf415246509365cee48c030d36 [term 1 FOLLOWER]: Refusing update from remote peer dc3142ac2c8548c48750d63e7eca5961: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:03:02.970036 14783 raft_consensus.cc:1261] T dbeeb9af3ace42c68c40c02a7ba33c3d P bb22bd13bfd8404aa9d02498577cd212 [term 1 FOLLOWER]: Refusing update from remote peer dc3142ac2c8548c48750d63e7eca5961: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:03:02.971426 14996 consensus_queue.cc:1035] T dbeeb9af3ace42c68c40c02a7ba33c3d P dc3142ac2c8548c48750d63e7eca5961 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb22bd13bfd8404aa9d02498577cd212" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 45731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:03:02.971951 14990 consensus_queue.cc:1035] T dbeeb9af3ace42c68c40c02a7ba33c3d P dc3142ac2c8548c48750d63e7eca5961 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ee2abdf415246509365cee48c030d36" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 32815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:03:03.006399 14502 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:40220:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 03:03:03.006779 14502 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:40220:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 03:03:03.011927 14502 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f53133eff48348659b64246f2f8317c5: Sending DeleteTablet for 3 replicas of tablet dbeeb9af3ace42c68c40c02a7ba33c3d
I20260806 03:03:03.013844 14763 tablet_service.cc:1514] Processing DeleteTablet for tablet dbeeb9af3ace42c68c40c02a7ba33c3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:03:03 UTC) from {username='kudu', principal='kudu/127.31.74.254@KRBTEST.COM'} at 127.0.0.1:48498
I20260806 03:03:03.014019 14899 tablet_service.cc:1514] Processing DeleteTablet for tablet dbeeb9af3ace42c68c40c02a7ba33c3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:03:03 UTC) from {username='kudu', principal='kudu/127.31.74.254@KRBTEST.COM'} at 127.0.0.1:54706
I20260806 03:03:03.014027 14627 tablet_service.cc:1514] Processing DeleteTablet for tablet dbeeb9af3ace42c68c40c02a7ba33c3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:03:03 UTC) from {username='kudu', principal='kudu/127.31.74.254@KRBTEST.COM'} at 127.0.0.1:55588
I20260806 03:03:03.020828 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 14542
I20260806 03:03:03.028918 15010 tablet_replica.cc:330] T dbeeb9af3ace42c68c40c02a7ba33c3d P bb22bd13bfd8404aa9d02498577cd212: stopping tablet replica
I20260806 03:03:03.029438 15010 raft_consensus.cc:2229] T dbeeb9af3ace42c68c40c02a7ba33c3d P bb22bd13bfd8404aa9d02498577cd212 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:03:03.029953 15010 raft_consensus.cc:2258] T dbeeb9af3ace42c68c40c02a7ba33c3d P bb22bd13bfd8404aa9d02498577cd212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:03:03.033234 15010 ts_tablet_manager.cc:1905] T dbeeb9af3ace42c68c40c02a7ba33c3d P bb22bd13bfd8404aa9d02498577cd212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:03:03.051549 15010 ts_tablet_manager.cc:1918] T dbeeb9af3ace42c68c40c02a7ba33c3d P bb22bd13bfd8404aa9d02498577cd212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 03:03:03.051826 15010 log.cc:1196] T dbeeb9af3ace42c68c40c02a7ba33c3d P bb22bd13bfd8404aa9d02498577cd212: Deleting WAL directory at /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-1/wal/wals/dbeeb9af3ace42c68c40c02a7ba33c3d
I20260806 03:03:03.052713 15010 ts_tablet_manager.cc:1939] T dbeeb9af3ace42c68c40c02a7ba33c3d P bb22bd13bfd8404aa9d02498577cd212: Deleting consensus metadata
I20260806 03:03:03.055826 14490 catalog_manager.cc:4841] TS bb22bd13bfd8404aa9d02498577cd212 (127.31.74.194:45731): tablet dbeeb9af3ace42c68c40c02a7ba33c3d (table test-table [id=7348ff3e3b54480cba9f8af688e83e62]) successfully deleted
I20260806 03:03:03.057514 15011 tablet_replica.cc:330] T dbeeb9af3ace42c68c40c02a7ba33c3d P 2ee2abdf415246509365cee48c030d36: stopping tablet replica
I20260806 03:03:03.058171 15011 raft_consensus.cc:2229] T dbeeb9af3ace42c68c40c02a7ba33c3d P 2ee2abdf415246509365cee48c030d36 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:03:03.058898 15011 raft_consensus.cc:2258] T dbeeb9af3ace42c68c40c02a7ba33c3d P 2ee2abdf415246509365cee48c030d36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:03:03.059748 14981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:03.023233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55614 (local address 127.31.74.193:37831)
0806 03:03:03.023546 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0806 03:03:03.023556 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 03:03:03.023586 (+ 30us) server_negotiation.cc:407] Connection header received
0806 03:03:03.023680 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:03:03.023689 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:03:03.023806 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:03:03.023945 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:03:03.032456 (+ 8511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:03.034099 (+ 1643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:03.036945 (+ 2846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:03.038060 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:03.038184 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:03:03.043646 (+ 5462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:03:03.043674 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:03:03.043703 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:03:03.043798 (+ 95us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:03:03.050859 (+ 7061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:03:03.053996 (+ 3137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:03:03.054009 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:03:03.054013 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:03:03.054325 (+ 312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:03:03.055409 (+ 1084us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:03:03.055417 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:03:03.055420 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:03:03.056531 (+ 1111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:03:03.058239 (+ 1708us) server_negotiation.cc:1035] Waiting for connection context
0806 03:03:03.058831 (+ 592us) server_negotiation.cc:299] Negotiation successful
0806 03:03:03.059189 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260806 03:03:03.060626 15012 tablet_replica.cc:330] T dbeeb9af3ace42c68c40c02a7ba33c3d P dc3142ac2c8548c48750d63e7eca5961: stopping tablet replica
I20260806 03:03:03.060714 14688 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 03:03:03.062306 15011 ts_tablet_manager.cc:1905] T dbeeb9af3ace42c68c40c02a7ba33c3d P 2ee2abdf415246509365cee48c030d36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:03:03.069818 15011 ts_tablet_manager.cc:1918] T dbeeb9af3ace42c68c40c02a7ba33c3d P 2ee2abdf415246509365cee48c030d36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 03:03:03.069996 15011 log.cc:1196] T dbeeb9af3ace42c68c40c02a7ba33c3d P 2ee2abdf415246509365cee48c030d36: Deleting WAL directory at /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-2/wal/wals/dbeeb9af3ace42c68c40c02a7ba33c3d
I20260806 03:03:03.070647 15011 ts_tablet_manager.cc:1939] T dbeeb9af3ace42c68c40c02a7ba33c3d P 2ee2abdf415246509365cee48c030d36: Deleting consensus metadata
I20260806 03:03:03.073046 14488 catalog_manager.cc:4841] TS 2ee2abdf415246509365cee48c030d36 (127.31.74.195:32815): tablet dbeeb9af3ace42c68c40c02a7ba33c3d (table test-table [id=7348ff3e3b54480cba9f8af688e83e62]) successfully deleted
I20260806 03:03:03.061131 15012 raft_consensus.cc:2229] T dbeeb9af3ace42c68c40c02a7ba33c3d P dc3142ac2c8548c48750d63e7eca5961 [term 1 LEADER]: Raft consensus shutting down.
I20260806 03:03:03.173276 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 14542
W20260806 03:03:03.203042 14487 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.74.193:37831 (error 108)
I20260806 03:03:03.203179 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 14697
I20260806 03:03:03.205590 14983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:03.204890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:37831 (local address 127.0.0.1:55616)
0806 03:03:03.205261 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:03.205431 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.193:37831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260806 03:03:03.206019 14487 catalog_manager.cc:4584] TS dc3142ac2c8548c48750d63e7eca5961 (127.31.74.193:37831): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dbeeb9af3ace42c68c40c02a7ba33c3d: Network error: Client connection negotiation failed: client connection to 127.31.74.193:37831: connect: Connection refused (error 111)
I20260806 03:03:03.222879 14982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:03.204346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48506 (local address 127.31.74.194:45731)
0806 03:03:03.204578 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0806 03:03:03.204583 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 03:03:03.204946 (+ 363us) server_negotiation.cc:407] Connection header received
0806 03:03:03.205169 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:03:03.205174 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:03:03.205277 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:03:03.205411 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:03:03.207312 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:03.208113 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:03.209550 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:03.209966 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:03.210119 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:03:03.214978 (+ 4859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:03:03.215000 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:03:03.215018 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:03:03.215072 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:03:03.219530 (+ 4458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:03:03.220494 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:03:03.220504 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:03:03.220506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:03:03.220653 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:03:03.221097 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:03:03.221103 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:03:03.221106 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:03:03.221527 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:03:03.221865 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0806 03:03:03.222198 (+ 333us) server_negotiation.cc:299] Negotiation successful
0806 03:03:03.222465 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260806 03:03:03.223654 14825 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 03:03:03.260157 14983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:03.259499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:37831 (local address 127.0.0.1:55618)
0806 03:03:03.259854 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:03.259976 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.193:37831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201}
I20260806 03:03:03.261472 14983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:03.261033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:37831 (local address 127.0.0.1:55626)
0806 03:03:03.261209 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:03.261300 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.193:37831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260806 03:03:03.322454 14983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:03.321842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:37831 (local address 127.0.0.1:55638)
0806 03:03:03.322117 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:03.322232 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.193:37831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260806 03:03:03.323850 14983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:03.323335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:37831 (local address 127.0.0.1:55644)
0806 03:03:03.323590 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:03.323685 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.193:37831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260806 03:03:03.333884 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 14697
I20260806 03:03:03.361568 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 14834
I20260806 03:03:03.381892 14980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:03.363087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54722 (local address 127.31.74.195:32815)
0806 03:03:03.363455 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0806 03:03:03.363461 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 03:03:03.363491 (+ 30us) server_negotiation.cc:407] Connection header received
0806 03:03:03.363585 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:03:03.363590 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:03:03.363706 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:03:03.363909 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:03:03.365535 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:03.366571 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:03.368517 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:03.369009 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:03.369127 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:03:03.372978 (+ 3851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:03:03.372998 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:03:03.373016 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:03:03.373066 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:03:03.377633 (+ 4567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:03:03.378813 (+ 1180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:03:03.378825 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:03:03.378831 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:03:03.379052 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:03:03.379860 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:03:03.379864 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:03:03.379866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:03:03.380302 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:03:03.380578 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0806 03:03:03.381235 (+ 657us) server_negotiation.cc:299] Negotiation successful
0806 03:03:03.381536 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 03:03:03.382787 14962 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 03:03:03.405274 14983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:03.404547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:37831 (local address 127.0.0.1:55660)
0806 03:03:03.404878 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:03.405019 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.193:37831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260806 03:03:03.406512 14983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:03.406066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:37831 (local address 127.0.0.1:55676)
0806 03:03:03.406295 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:03.406392 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.193:37831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260806 03:03:03.492455 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 14834
I20260806 03:03:03.514732 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 14469
I20260806 03:03:03.516148 14535 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 03:03:03.647995 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 14469
I20260806 03:03:03.648121 14983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:03.647354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:37831 (local address 127.0.0.1:55678)
0806 03:03:03.647716 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:03.647861 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.193:37831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195}
I20260806 03:03:03.680896 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:33865
--webserver_interface=127.31.74.254
--webserver_port=39355
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab
--principal=kudu/127.31.74.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:44989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.74.254:33865
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:03:04.215023 15018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:03:04.216523 15018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:03:04.216733 15018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:03:04.226603 15018 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:03:04.226763 15018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:03:04.226820 15018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:03:04.226925 15018 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:03:04.227231 15018 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:03:04.241389 15018 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:44989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.74.254:33865
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab
--principal=kudu/127.31.74.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:33865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.74.254
--webserver_port=39355
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:03:04.242981 15018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:03:04.244477 15018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:03:04.266376 15026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:03:04.266651 15028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:03:04.266883 15025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:03:04.267405 15018 server_base.cc:1034] running on GCE node
I20260806 03:03:04.268544 15018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:03:04.270231 15018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:03:04.271489 15018 hybrid_clock.cc:648] HybridClock initialized: now 1785985384271400 us; error 78 us; skew 500 ppm
I20260806 03:03:04.271867 15018 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 03:03:04 dist-test-slave-dhph krb5kdc[13889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985384, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:03:04.275605 15018 init.cc:377] Logged in from keytab as kudu/127.31.74.254@KRBTEST.COM (short username kudu)
I20260806 03:03:04.278947 15018 webserver.cc:460] Webserver started at http://127.31.74.254:39355/ using document root <none> and password file <none>
I20260806 03:03:04.280227 15018 fs_manager.cc:359] Metadata directory not provided
I20260806 03:03:04.280426 15018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:03:04.290490 15018 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.005s sys 0.000s
I20260806 03:03:04.294413 15036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:03:04.296331 15018 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 03:03:04.296671 15018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/master-0/wal
uuid: "f53133eff48348659b64246f2f8317c5"
format_stamp: "Formatted at 2026-08-06 03:02:56 on dist-test-slave-dhph"
server_key: "245a2d316f63d43e223222235a5005cc"
server_key_iv: "b2b761d8fb8444f5ee18e31d55c49e36"
server_key_version: "encryptionkey@0"
I20260806 03:03:04.298344 15018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:03:04.361033 15018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:03:04.368501 15018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:03:04.368886 15018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:03:04.408986 15018 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.254:33865
I20260806 03:03:04.408999 15089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.254:33865 every 8 connection(s)
I20260806 03:03:04.410975 15018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
I20260806 03:03:04.413636 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 15018
I20260806 03:03:04.414522 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.193:37831
--local_ip_for_outbound_sockets=127.31.74.193
--tserver_master_addrs=127.31.74.254:33865
--webserver_port=37087
--webserver_interface=127.31.74.193
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.193.keytab
--principal=kudu/127.31.74.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:44989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 03:03:04.420840 15090 sys_catalog.cc:263] Verifying existing consensus state
I20260806 03:03:04.425098 15090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f53133eff48348659b64246f2f8317c5: Bootstrap starting.
I20260806 03:03:04.472393 15090 log.cc:824] T 00000000000000000000000000000000 P f53133eff48348659b64246f2f8317c5: Log is configured to *not* fsync() on all Append() calls
I20260806 03:03:04.505999 15090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f53133eff48348659b64246f2f8317c5: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 03:03:04.506928 15090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f53133eff48348659b64246f2f8317c5: Bootstrap complete.
I20260806 03:03:04.527161 15090 raft_consensus.cc:348] T 00000000000000000000000000000000 P f53133eff48348659b64246f2f8317c5 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f53133eff48348659b64246f2f8317c5" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 33865 } }
I20260806 03:03:04.530313 15090 raft_consensus.cc:729] T 00000000000000000000000000000000 P f53133eff48348659b64246f2f8317c5 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: f53133eff48348659b64246f2f8317c5, State: Initialized, Role: FOLLOWER
I20260806 03:03:04.531790 15090 consensus_queue.cc:260] T 00000000000000000000000000000000 P f53133eff48348659b64246f2f8317c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f53133eff48348659b64246f2f8317c5" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 33865 } }
I20260806 03:03:04.532438 15090 raft_consensus.cc:388] T 00000000000000000000000000000000 P f53133eff48348659b64246f2f8317c5 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:03:04.532640 15090 raft_consensus.cc:482] T 00000000000000000000000000000000 P f53133eff48348659b64246f2f8317c5 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:03:04.532895 15090 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f53133eff48348659b64246f2f8317c5 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:03:04.542457 15090 raft_consensus.cc:504] T 00000000000000000000000000000000 P f53133eff48348659b64246f2f8317c5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f53133eff48348659b64246f2f8317c5" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 33865 } }
I20260806 03:03:04.543308 15090 leader_election.cc:302] T 00000000000000000000000000000000 P f53133eff48348659b64246f2f8317c5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f53133eff48348659b64246f2f8317c5; no voters:
I20260806 03:03:04.544727 15090 leader_election.cc:288] T 00000000000000000000000000000000 P f53133eff48348659b64246f2f8317c5 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260806 03:03:04.545023 15094 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f53133eff48348659b64246f2f8317c5 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:03:04.546829 15094 raft_consensus.cc:686] T 00000000000000000000000000000000 P f53133eff48348659b64246f2f8317c5 [term 3 LEADER]: Becoming Leader. State: Replica: f53133eff48348659b64246f2f8317c5, State: Running, Role: LEADER
I20260806 03:03:04.547973 15090 sys_catalog.cc:564] T 00000000000000000000000000000000 P f53133eff48348659b64246f2f8317c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:03:04.547901 15094 consensus_queue.cc:237] T 00000000000000000000000000000000 P f53133eff48348659b64246f2f8317c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f53133eff48348659b64246f2f8317c5" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 33865 } }
I20260806 03:03:04.551889 15096 sys_catalog.cc:455] T 00000000000000000000000000000000 P f53133eff48348659b64246f2f8317c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f53133eff48348659b64246f2f8317c5. Latest consensus state: current_term: 3 leader_uuid: "f53133eff48348659b64246f2f8317c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f53133eff48348659b64246f2f8317c5" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 33865 } } }
I20260806 03:03:04.551910 15095 sys_catalog.cc:455] T 00000000000000000000000000000000 P f53133eff48348659b64246f2f8317c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "f53133eff48348659b64246f2f8317c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f53133eff48348659b64246f2f8317c5" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 33865 } } }
I20260806 03:03:04.552418 15095 sys_catalog.cc:458] T 00000000000000000000000000000000 P f53133eff48348659b64246f2f8317c5 [sys.catalog]: This master's current role is: LEADER
I20260806 03:03:04.552415 15096 sys_catalog.cc:458] T 00000000000000000000000000000000 P f53133eff48348659b64246f2f8317c5 [sys.catalog]: This master's current role is: LEADER
I20260806 03:03:04.556546 15100 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:03:04.570899 15100 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:03:04.572717 15100 catalog_manager.cc:1259] Loaded cluster ID: 2cd1e9d5e7e44de1b93b1f61ecaf923a
I20260806 03:03:04.572944 15100 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260806 03:03:04.575673 15100 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f7737c119ad google::LogMessage::Fail() at ??:0
@ 0x7f7737c1488a google::LogMessage::SendToLog() at ??:0
@ 0x7f7737c114de google::LogMessage::Flush() at ??:0
@ 0x7f7737c12a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
I20260806 03:03:04.594388 15113 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7f774d644512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f774d641976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f774d6c9610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f774d6c9439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f773a85d345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f773a86091c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f773a860799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f773a83beaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f773db156db start_thread at ??:0
@ 0x7f7736aab61f clone at ??:0
W20260806 03:03:05.001305 15092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:03:05.001706 15092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:03:05.001893 15092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:03:05.012542 15092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:03:05.012876 15092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:03:05.013095 15092 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.193
I20260806 03:03:05.025111 15092 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:44989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.193.keytab
--principal=kudu/127.31.74.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.193:37831
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.74.193
--webserver_port=37087
--tserver_master_addrs=127.31.74.254:33865
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.193
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:03:05.026691 15092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:03:05.028529 15092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:03:05.045606 15135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:03:05.045573 15138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:03:05.045528 15136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:03:05.047607 15092 server_base.cc:1034] running on GCE node
I20260806 03:03:05.048473 15092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:03:05.049718 15092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:03:05.051034 15092 hybrid_clock.cc:648] HybridClock initialized: now 1785985385050952 us; error 74 us; skew 500 ppm
I20260806 03:03:05.051764 15092 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 03:03:05 dist-test-slave-dhph krb5kdc[13889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985385, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:03:05.064129 15092 init.cc:377] Logged in from keytab as kudu/127.31.74.193@KRBTEST.COM (short username kudu)
I20260806 03:03:05.066145 15092 webserver.cc:460] Webserver started at http://127.31.74.193:37087/ using document root <none> and password file <none>
I20260806 03:03:05.067065 15092 fs_manager.cc:359] Metadata directory not provided
I20260806 03:03:05.067235 15092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:03:05.075363 15092 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.003s
I20260806 03:03:05.080756 15145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:03:05.082866 15092 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.001s
I20260806 03:03:05.083098 15092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-0/wal
uuid: "dc3142ac2c8548c48750d63e7eca5961"
format_stamp: "Formatted at 2026-08-06 03:02:57 on dist-test-slave-dhph"
server_key: "60ac3b964d89f11416c30d1e25045229"
server_key_iv: "8bc41061c44f56028dd17211372facb3"
server_key_version: "encryptionkey@0"
I20260806 03:03:05.084229 15092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:03:05.100746 15092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:03:05.106216 15092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:03:05.106680 15092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:03:05.108140 15092 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:03:05.110941 15152 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260806 03:03:05.117282 15092 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260806 03:03:05.117656 15092 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.000s sys 0.001s
I20260806 03:03:05.117841 15092 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260806 03:03:05.121497 15092 ts_tablet_manager.cc:610] Registered 1 tablets
I20260806 03:03:05.121641 15092 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.000s sys 0.001s
I20260806 03:03:05.122515 15152 tablet_bootstrap.cc:492] T dbeeb9af3ace42c68c40c02a7ba33c3d P dc3142ac2c8548c48750d63e7eca5961: Bootstrap starting.
I20260806 03:03:05.173368 15152 log.cc:824] T dbeeb9af3ace42c68c40c02a7ba33c3d P dc3142ac2c8548c48750d63e7eca5961: Log is configured to *not* fsync() on all Append() calls
I20260806 03:03:05.192623 15152 tablet_bootstrap.cc:492] T dbeeb9af3ace42c68c40c02a7ba33c3d P dc3142ac2c8548c48750d63e7eca5961: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 03:03:05.193436 15152 tablet_bootstrap.cc:492] T dbeeb9af3ace42c68c40c02a7ba33c3d P dc3142ac2c8548c48750d63e7eca5961: Bootstrap complete.
I20260806 03:03:05.196177 15152 ts_tablet_manager.cc:1397] T dbeeb9af3ace42c68c40c02a7ba33c3d P dc3142ac2c8548c48750d63e7eca5961: Time spent bootstrapping tablet: real 0.074s user 0.050s sys 0.020s
I20260806 03:03:05.199476 15092 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.193:37831
I20260806 03:03:05.199499 15264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.193:37831 every 8 connection(s)
I20260806 03:03:05.201349 15092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
I20260806 03:03:05.205717 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 15092
I20260806 03:03:05.206804 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.194:45731
--local_ip_for_outbound_sockets=127.31.74.194
--tserver_master_addrs=127.31.74.254:33865
--webserver_port=42495
--webserver_interface=127.31.74.194
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.194.keytab
--principal=kudu/127.31.74.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:44989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 03:03:05.209494 15267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:05.205196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:33865 (local address 127.31.74.193:42675)
0806 03:03:05.206714 (+ 1518us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:05.207152 (+ 438us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1055,"thread_start_us":665,"threads_started":1}
W20260806 03:03:05.211424 15265 heartbeater.cc:644] Failed to heartbeat to 127.31.74.254:33865 (0 consecutive failures): Network error: Failed to ping master at 127.31.74.254:33865: Client connection negotiation failed: client connection to 127.31.74.254:33865: connect: Connection refused (error 111)
I20260806 03:03:05.210711 15152 raft_consensus.cc:348] T dbeeb9af3ace42c68c40c02a7ba33c3d P dc3142ac2c8548c48750d63e7eca5961 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ee2abdf415246509365cee48c030d36" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 32815 } } peers { permanent_uuid: "dc3142ac2c8548c48750d63e7eca5961" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37831 } } peers { permanent_uuid: "bb22bd13bfd8404aa9d02498577cd212" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 45731 } }
I20260806 03:03:05.213284 15152 raft_consensus.cc:729] T dbeeb9af3ace42c68c40c02a7ba33c3d P dc3142ac2c8548c48750d63e7eca5961 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc3142ac2c8548c48750d63e7eca5961, State: Initialized, Role: FOLLOWER
I20260806 03:03:05.213814 15267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:05.213150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:33865 (local address 127.31.74.193:49461)
0806 03:03:05.213494 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:05.213622 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260806 03:03:05.214376 15152 consensus_queue.cc:260] T dbeeb9af3ace42c68c40c02a7ba33c3d P dc3142ac2c8548c48750d63e7eca5961 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ee2abdf415246509365cee48c030d36" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 32815 } } peers { permanent_uuid: "dc3142ac2c8548c48750d63e7eca5961" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37831 } } peers { permanent_uuid: "bb22bd13bfd8404aa9d02498577cd212" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 45731 } }
I20260806 03:03:05.216722 15267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:05.215386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:33865 (local address 127.31.74.193:59049)
0806 03:03:05.215858 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:05.216414 (+ 556us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303}
W20260806 03:03:05.217413 15265 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 03:03:05.218202 15152 ts_tablet_manager.cc:1428] T dbeeb9af3ace42c68c40c02a7ba33c3d P dc3142ac2c8548c48750d63e7eca5961: Time spent starting tablet: real 0.022s user 0.016s sys 0.004s
I20260806 03:03:05.218757 15267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:05.218129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:33865 (local address 127.31.74.193:41813)
0806 03:03:05.218389 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:05.218506 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260806 03:03:05.220621 15267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:05.220113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:33865 (local address 127.31.74.193:38491)
0806 03:03:05.220331 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:05.220434 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260806 03:03:05.649487 15269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:03:05.649996 15269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:03:05.650164 15269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:03:05.660846 15269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:03:05.661022 15269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:03:05.661158 15269 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.194
I20260806 03:03:05.674310 15269 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:44989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.194.keytab
--principal=kudu/127.31.74.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.194:45731
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.74.194
--webserver_port=42495
--tserver_master_addrs=127.31.74.254:33865
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.194
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:03:05.675805 15269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:03:05.677659 15269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:03:05.696858 15278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:03:05.696861 15274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:03:05.698283 15276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:03:05.698894 15269 server_base.cc:1034] running on GCE node
I20260806 03:03:05.699643 15269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:03:05.700757 15269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:03:05.702049 15269 hybrid_clock.cc:648] HybridClock initialized: now 1785985385701968 us; error 62 us; skew 500 ppm
I20260806 03:03:05.702710 15269 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 03:03:05 dist-test-slave-dhph krb5kdc[13889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985385, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:03:05.706748 15269 init.cc:377] Logged in from keytab as kudu/127.31.74.194@KRBTEST.COM (short username kudu)
I20260806 03:03:05.708691 15269 webserver.cc:460] Webserver started at http://127.31.74.194:42495/ using document root <none> and password file <none>
I20260806 03:03:05.709811 15269 fs_manager.cc:359] Metadata directory not provided
I20260806 03:03:05.709946 15269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:03:05.719401 15269 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.007s sys 0.000s
I20260806 03:03:05.724284 15285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:03:05.726199 15269 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260806 03:03:05.726425 15269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-1/wal
uuid: "bb22bd13bfd8404aa9d02498577cd212"
format_stamp: "Formatted at 2026-08-06 03:02:57 on dist-test-slave-dhph"
server_key: "3afd3b3c8bda75f0f7e6d1aaa9388e51"
server_key_iv: "493b446ae358887437b3a868469f5f6d"
server_key_version: "encryptionkey@0"
I20260806 03:03:05.727944 15269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:03:05.757566 15269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:03:05.764159 15269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:03:05.764552 15269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:03:05.765946 15269 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:03:05.767755 15269 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:03:05.767872 15269 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:03:05.768107 15269 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:03:05.768245 15269 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:03:05.811419 15269 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.194:45731
I20260806 03:03:05.811524 15401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.194:45731 every 8 connection(s)
I20260806 03:03:05.813400 15269 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
I20260806 03:03:05.820330 15404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:05.817326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:33865 (local address 127.31.74.194:37633)
0806 03:03:05.818882 (+ 1556us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:05.819152 (+ 270us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1012,"thread_start_us":547,"threads_started":1}
W20260806 03:03:05.821249 15402 heartbeater.cc:644] Failed to heartbeat to 127.31.74.254:33865 (0 consecutive failures): Network error: Failed to ping master at 127.31.74.254:33865: Client connection negotiation failed: client connection to 127.31.74.254:33865: connect: Connection refused (error 111)
I20260806 03:03:05.822111 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 15269
I20260806 03:03:05.822675 15404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:05.822160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:33865 (local address 127.31.74.194:55747)
0806 03:03:05.822391 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:05.822477 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260806 03:03:05.822952 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.195:32815
--local_ip_for_outbound_sockets=127.31.74.195
--tserver_master_addrs=127.31.74.254:33865
--webserver_port=35447
--webserver_interface=127.31.74.195
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.195.keytab
--principal=kudu/127.31.74.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:44989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 03:03:05.823791 15404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:05.823394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:33865 (local address 127.31.74.194:45327)
0806 03:03:05.823559 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:05.823658 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260806 03:03:05.824103 15402 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 03:03:06.223729 15406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:06.222362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:33865 (local address 127.31.74.193:35543)
0806 03:03:06.223340 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:06.223476 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":834,"thread_start_us":646,"threads_started":1}
W20260806 03:03:06.255981 15405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:03:06.256356 15405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:03:06.256548 15405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:03:06.264971 15405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:03:06.265139 15405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:03:06.265272 15405 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.195
I20260806 03:03:06.277047 15405 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:44989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.195.keytab
--principal=kudu/127.31.74.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.195:32815
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.74.195
--webserver_port=35447
--tserver_master_addrs=127.31.74.254:33865
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.195
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:03:06.278550 15405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:03:06.280182 15405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:03:06.295459 15411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:03:06.295787 15414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:03:06.296061 15412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:03:06.296274 15405 server_base.cc:1034] running on GCE node
I20260806 03:03:06.297039 15405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:03:06.298177 15405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:03:06.299494 15405 hybrid_clock.cc:648] HybridClock initialized: now 1785985386299446 us; error 36 us; skew 500 ppm
I20260806 03:03:06.299896 15405 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 03:03:06 dist-test-slave-dhph krb5kdc[13889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985386, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:03:06.304277 15405 init.cc:377] Logged in from keytab as kudu/127.31.74.195@KRBTEST.COM (short username kudu)
I20260806 03:03:06.306072 15405 webserver.cc:460] Webserver started at http://127.31.74.195:35447/ using document root <none> and password file <none>
I20260806 03:03:06.306944 15405 fs_manager.cc:359] Metadata directory not provided
I20260806 03:03:06.307296 15405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:03:06.315127 15405 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20260806 03:03:06.318979 15421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:03:06.320793 15405 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 03:03:06.320991 15405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-2/wal
uuid: "2ee2abdf415246509365cee48c030d36"
format_stamp: "Formatted at 2026-08-06 03:02:58 on dist-test-slave-dhph"
server_key: "ec3cee2c35ef448d2ed30c57459f5885"
server_key_iv: "c75b10fff81f46f9977292ab6a2b8c6e"
server_key_version: "encryptionkey@0"
I20260806 03:03:06.322196 15405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:03:06.355089 15405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:03:06.360505 15405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:03:06.360795 15405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:03:06.362677 15405 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:03:06.365495 15405 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:03:06.365677 15405 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:03:06.365818 15405 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:03:06.365931 15405 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:03:06.411293 15405 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.195:32815
I20260806 03:03:06.411398 15537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.195:32815 every 8 connection(s)
I20260806 03:03:06.413120 15405 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
I20260806 03:03:06.413995 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 15405
I20260806 03:03:06.419238 15542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:06.416522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:33865 (local address 127.31.74.195:59033)
0806 03:03:06.417356 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:06.417588 (+ 232us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":348,"threads_started":1}
W20260806 03:03:06.420429 15538 heartbeater.cc:644] Failed to heartbeat to 127.31.74.254:33865 (0 consecutive failures): Network error: Failed to ping master at 127.31.74.254:33865: Client connection negotiation failed: client connection to 127.31.74.254:33865: connect: Connection refused (error 111)
I20260806 03:03:06.421947 15542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:06.421322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:33865 (local address 127.31.74.195:46211)
0806 03:03:06.421634 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:06.421755 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
I20260806 03:03:06.423892 15542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:06.423326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:33865 (local address 127.31.74.195:59497)
0806 03:03:06.423561 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:06.423667 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
W20260806 03:03:06.424420 15538 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 03:03:06.827298 15543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:06.825232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:33865 (local address 127.31.74.194:59107)
0806 03:03:06.826742 (+ 1510us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:06.826879 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1299,"thread_start_us":503,"threads_started":1}
I20260806 03:03:07.088817 15545 raft_consensus.cc:482] T dbeeb9af3ace42c68c40c02a7ba33c3d P dc3142ac2c8548c48750d63e7eca5961 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:03:07.089149 15545 raft_consensus.cc:504] T dbeeb9af3ace42c68c40c02a7ba33c3d P dc3142ac2c8548c48750d63e7eca5961 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ee2abdf415246509365cee48c030d36" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 32815 } } peers { permanent_uuid: "dc3142ac2c8548c48750d63e7eca5961" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37831 } } peers { permanent_uuid: "bb22bd13bfd8404aa9d02498577cd212" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 45731 } }
I20260806 03:03:07.090869 15545 leader_election.cc:288] T dbeeb9af3ace42c68c40c02a7ba33c3d P dc3142ac2c8548c48750d63e7eca5961 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2ee2abdf415246509365cee48c030d36 (127.31.74.195:32815), bb22bd13bfd8404aa9d02498577cd212 (127.31.74.194:45731)
Aug 06 03:03:07 dist-test-slave-dhph krb5kdc[13889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985385, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.193@KRBTEST.COM for kudu/127.31.74.194@KRBTEST.COM
Aug 06 03:03:07 dist-test-slave-dhph krb5kdc[13889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985385, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.193@KRBTEST.COM for kudu/127.31.74.195@KRBTEST.COM
I20260806 03:03:07.131913 15546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:07.091494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.194:45731 (local address 127.31.74.193:56799)
0806 03:03:07.104658 (+ 13164us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:07.104677 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 03:03:07.105176 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:03:07.108041 (+ 2865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:03:07.108054 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:03:07.109107 (+ 1053us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:03:07.110230 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:03:07.110262 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:03:07.112841 (+ 2579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:03:07.112846 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:03:07.114432 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:03:07.114446 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:03:07.115567 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:03:07.115575 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:03:07.115916 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:03:07.116834 (+ 918us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:03:07.116852 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:03:07.119718 (+ 2866us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:03:07.125216 (+ 5498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:03:07.125228 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:03:07.125244 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:03:07.125867 (+ 623us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:03:07.126916 (+ 1049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:03:07.126922 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:03:07.126926 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:03:07.127304 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:03:07.129617 (+ 2313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:03:07.129628 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:03:07.130111 (+ 483us) client_negotiation.cc:769] Sending connection context
0806 03:03:07.130453 (+ 342us) client_negotiation.cc:240] Negotiation successful
0806 03:03:07.131115 (+ 662us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13018,"mutex_wait_us":157,"spinlock_wait_cycles":64128,"thread_start_us":12550,"threads_started":1}
I20260806 03:03:07.132185 15549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:07.091813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.193:56799 (local address 127.31.74.194:45731)
0806 03:03:07.102861 (+ 11048us) server_negotiation.cc:206] Beginning negotiation
0806 03:03:07.102893 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0806 03:03:07.104840 (+ 1947us) server_negotiation.cc:407] Connection header received
0806 03:03:07.107187 (+ 2347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:03:07.107197 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:03:07.107560 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:03:07.107894 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:03:07.110495 (+ 2601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:07.112572 (+ 2077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:07.114633 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:07.115323 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:07.115547 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:03:07.120091 (+ 4544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:03:07.120101 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:03:07.120116 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:03:07.120161 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:03:07.124933 (+ 4772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:03:07.126078 (+ 1145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:03:07.126096 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:03:07.126102 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:03:07.126318 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:03:07.127464 (+ 1146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:03:07.127471 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:03:07.127475 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:03:07.129442 (+ 1967us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:03:07.129677 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0806 03:03:07.130785 (+ 1108us) server_negotiation.cc:299] Negotiation successful
0806 03:03:07.131523 (+ 738us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10641,"thread_start_us":11132,"threads_started":1}
I20260806 03:03:07.133176 15547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:07.091143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.195:32815 (local address 127.31.74.193:43095)
0806 03:03:07.104465 (+ 13322us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:07.104513 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 03:03:07.105146 (+ 633us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:03:07.108041 (+ 2895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:03:07.108053 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:03:07.109182 (+ 1129us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:03:07.110312 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:03:07.110331 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:03:07.112838 (+ 2507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:03:07.112843 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:03:07.114428 (+ 1585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:03:07.114444 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:03:07.115589 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:03:07.115597 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:03:07.115911 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:03:07.116779 (+ 868us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:03:07.116811 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:03:07.119921 (+ 3110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:03:07.127782 (+ 7861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:03:07.127787 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:03:07.127791 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:03:07.128871 (+ 1080us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:03:07.130164 (+ 1293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:03:07.130168 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:03:07.130171 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:03:07.130337 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:03:07.131336 (+ 999us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:03:07.131342 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:03:07.131511 (+ 169us) client_negotiation.cc:769] Sending connection context
0806 03:03:07.131671 (+ 160us) client_negotiation.cc:240] Negotiation successful
0806 03:03:07.132012 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13159,"mutex_wait_us":267,"spinlock_wait_cycles":4480,"thread_start_us":13124,"threads_started":1}
I20260806 03:03:07.133399 15548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:07.091450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.193:43095 (local address 127.31.74.195:32815)
0806 03:03:07.103878 (+ 12428us) server_negotiation.cc:206] Beginning negotiation
0806 03:03:07.103913 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0806 03:03:07.104695 (+ 782us) server_negotiation.cc:407] Connection header received
0806 03:03:07.107041 (+ 2346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:03:07.107050 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:03:07.107496 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:03:07.107858 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:03:07.110548 (+ 2690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:07.112573 (+ 2025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:07.114644 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:07.115342 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:07.115567 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:03:07.120608 (+ 5041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:03:07.120621 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:03:07.120642 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:03:07.120708 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:03:07.127598 (+ 6890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:03:07.129346 (+ 1748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:03:07.129354 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:03:07.129361 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:03:07.129976 (+ 615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:03:07.130465 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:03:07.130470 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:03:07.130473 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:03:07.131069 (+ 596us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:03:07.131392 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0806 03:03:07.131931 (+ 539us) server_negotiation.cc:299] Negotiation successful
0806 03:03:07.132834 (+ 903us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12163,"thread_start_us":11994,"threads_started":1}
W20260806 03:03:07.137732 15149 leader_election.cc:341] T dbeeb9af3ace42c68c40c02a7ba33c3d P dc3142ac2c8548c48750d63e7eca5961 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bb22bd13bfd8404aa9d02498577cd212 (127.31.74.194:45731): Not found: Tablet not found: dbeeb9af3ace42c68c40c02a7ba33c3d
W20260806 03:03:07.138104 15147 leader_election.cc:341] T dbeeb9af3ace42c68c40c02a7ba33c3d P dc3142ac2c8548c48750d63e7eca5961 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2ee2abdf415246509365cee48c030d36 (127.31.74.195:32815): Not found: Tablet not found: dbeeb9af3ace42c68c40c02a7ba33c3d
I20260806 03:03:07.138265 15147 leader_election.cc:302] T dbeeb9af3ace42c68c40c02a7ba33c3d P dc3142ac2c8548c48750d63e7eca5961 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc3142ac2c8548c48750d63e7eca5961; no voters: 2ee2abdf415246509365cee48c030d36, bb22bd13bfd8404aa9d02498577cd212
I20260806 03:03:07.138845 15545 raft_consensus.cc:2726] T dbeeb9af3ace42c68c40c02a7ba33c3d P dc3142ac2c8548c48750d63e7eca5961 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:03:07.225400 15547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:07.224888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:33865 (local address 127.31.74.193:48661)
0806 03:03:07.225094 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:07.225229 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260806 03:03:07.427541 15550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:07.425258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:33865 (local address 127.31.74.195:38739)
0806 03:03:07.426139 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:07.426464 (+ 325us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":706,"thread_start_us":478,"threads_started":1}
I20260806 03:03:07.829808 15551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:07.828854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:33865 (local address 127.31.74.194:43809)
0806 03:03:07.829451 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:07.829574 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":314,"threads_started":1}
I20260806 03:03:08.228098 15552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:08.226906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:33865 (local address 127.31.74.193:53189)
0806 03:03:08.227785 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:08.227889 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":747,"thread_start_us":500,"threads_started":1}
I20260806 03:03:08.430051 15553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:08.429221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:33865 (local address 127.31.74.195:36011)
0806 03:03:08.429741 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:08.429854 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":276,"threads_started":1}
I20260806 03:03:08.832547 15554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:08.831357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:33865 (local address 127.31.74.194:46955)
0806 03:03:08.832223 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:08.832331 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":742,"thread_start_us":427,"threads_started":1}
I20260806 03:03:09.231353 15555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:09.229897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:33865 (local address 127.31.74.193:33771)
0806 03:03:09.230737 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:09.230906 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":351,"threads_started":1}
I20260806 03:03:09.374734 15556 raft_consensus.cc:482] T dbeeb9af3ace42c68c40c02a7ba33c3d P dc3142ac2c8548c48750d63e7eca5961 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:03:09.375331 15556 raft_consensus.cc:504] T dbeeb9af3ace42c68c40c02a7ba33c3d P dc3142ac2c8548c48750d63e7eca5961 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ee2abdf415246509365cee48c030d36" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 32815 } } peers { permanent_uuid: "dc3142ac2c8548c48750d63e7eca5961" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37831 } } peers { permanent_uuid: "bb22bd13bfd8404aa9d02498577cd212" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 45731 } }
I20260806 03:03:09.376922 15556 leader_election.cc:288] T dbeeb9af3ace42c68c40c02a7ba33c3d P dc3142ac2c8548c48750d63e7eca5961 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2ee2abdf415246509365cee48c030d36 (127.31.74.195:32815), bb22bd13bfd8404aa9d02498577cd212 (127.31.74.194:45731)
W20260806 03:03:09.378268 15149 leader_election.cc:341] T dbeeb9af3ace42c68c40c02a7ba33c3d P dc3142ac2c8548c48750d63e7eca5961 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bb22bd13bfd8404aa9d02498577cd212 (127.31.74.194:45731): Not found: Tablet not found: dbeeb9af3ace42c68c40c02a7ba33c3d
W20260806 03:03:09.378539 15147 leader_election.cc:341] T dbeeb9af3ace42c68c40c02a7ba33c3d P dc3142ac2c8548c48750d63e7eca5961 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2ee2abdf415246509365cee48c030d36 (127.31.74.195:32815): Not found: Tablet not found: dbeeb9af3ace42c68c40c02a7ba33c3d
I20260806 03:03:09.378724 15147 leader_election.cc:302] T dbeeb9af3ace42c68c40c02a7ba33c3d P dc3142ac2c8548c48750d63e7eca5961 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc3142ac2c8548c48750d63e7eca5961; no voters: 2ee2abdf415246509365cee48c030d36, bb22bd13bfd8404aa9d02498577cd212
I20260806 03:03:09.379105 15556 raft_consensus.cc:2726] T dbeeb9af3ace42c68c40c02a7ba33c3d P dc3142ac2c8548c48750d63e7eca5961 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:03:09.432153 15557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:09.431291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:33865 (local address 127.31.74.195:55019)
0806 03:03:09.431861 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:09.431981 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":272,"threads_started":1}
I20260806 03:03:09.834731 15558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:09.833718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:33865 (local address 127.31.74.194:45929)
0806 03:03:09.834350 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:09.834471 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":358,"threads_started":1}
I20260806 03:03:10.234244 15559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:10.232930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:33865 (local address 127.31.74.193:53751)
0806 03:03:10.233665 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:10.233968 (+ 303us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":429,"threads_started":1}
I20260806 03:03:10.434863 15560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:10.433679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:33865 (local address 127.31.74.195:52199)
0806 03:03:10.434417 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:10.434517 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":393,"threads_started":1}
I20260806 03:03:10.836576 15561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:10.835754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:33865 (local address 127.31.74.194:36753)
0806 03:03:10.836295 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:10.836390 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":285,"threads_started":1}
I20260806 03:03:11.236528 15562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:11.235608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:33865 (local address 127.31.74.193:50935)
0806 03:03:11.236241 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:11.236339 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":297,"threads_started":1}
I20260806 03:03:11.437003 15563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:11.436087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:33865 (local address 127.31.74.195:38155)
0806 03:03:11.436689 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:11.436804 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":363,"threads_started":1}
I20260806 03:03:11.530514 15564 raft_consensus.cc:482] T dbeeb9af3ace42c68c40c02a7ba33c3d P dc3142ac2c8548c48750d63e7eca5961 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:03:11.530903 15564 raft_consensus.cc:504] T dbeeb9af3ace42c68c40c02a7ba33c3d P dc3142ac2c8548c48750d63e7eca5961 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ee2abdf415246509365cee48c030d36" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 32815 } } peers { permanent_uuid: "dc3142ac2c8548c48750d63e7eca5961" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37831 } } peers { permanent_uuid: "bb22bd13bfd8404aa9d02498577cd212" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 45731 } }
I20260806 03:03:11.531742 15564 leader_election.cc:288] T dbeeb9af3ace42c68c40c02a7ba33c3d P dc3142ac2c8548c48750d63e7eca5961 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2ee2abdf415246509365cee48c030d36 (127.31.74.195:32815), bb22bd13bfd8404aa9d02498577cd212 (127.31.74.194:45731)
W20260806 03:03:11.533052 15149 leader_election.cc:341] T dbeeb9af3ace42c68c40c02a7ba33c3d P dc3142ac2c8548c48750d63e7eca5961 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bb22bd13bfd8404aa9d02498577cd212 (127.31.74.194:45731): Not found: Tablet not found: dbeeb9af3ace42c68c40c02a7ba33c3d
W20260806 03:03:11.533277 15147 leader_election.cc:341] T dbeeb9af3ace42c68c40c02a7ba33c3d P dc3142ac2c8548c48750d63e7eca5961 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2ee2abdf415246509365cee48c030d36 (127.31.74.195:32815): Not found: Tablet not found: dbeeb9af3ace42c68c40c02a7ba33c3d
I20260806 03:03:11.533392 15147 leader_election.cc:302] T dbeeb9af3ace42c68c40c02a7ba33c3d P dc3142ac2c8548c48750d63e7eca5961 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc3142ac2c8548c48750d63e7eca5961; no voters: 2ee2abdf415246509365cee48c030d36, bb22bd13bfd8404aa9d02498577cd212
I20260806 03:03:11.533960 15564 raft_consensus.cc:2726] T dbeeb9af3ace42c68c40c02a7ba33c3d P dc3142ac2c8548c48750d63e7eca5961 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:03:11.839537 15565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:11.838192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:33865 (local address 127.31.74.194:49073)
0806 03:03:11.839167 (+ 975us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:11.839295 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":837,"thread_start_us":550,"threads_started":1}
I20260806 03:03:12.240336 15566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:12.237820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:33865 (local address 127.31.74.193:53279)
0806 03:03:12.238888 (+ 1068us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:12.239040 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":266,"threads_started":1}
I20260806 03:03:12.439467 15567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:12.438375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:33865 (local address 127.31.74.195:56743)
0806 03:03:12.439095 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:12.439223 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":414,"threads_started":1}
I20260806 03:03:12.842882 15568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:12.840927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:33865 (local address 127.31.74.194:40161)
0806 03:03:12.841993 (+ 1066us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:12.842340 (+ 347us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":766,"thread_start_us":367,"threads_started":1}
I20260806 03:03:13.242841 15569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:13.241858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:33865 (local address 127.31.74.193:42887)
0806 03:03:13.242474 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:13.242577 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":346,"threads_started":1}
I20260806 03:03:13.442173 15570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:13.440880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:33865 (local address 127.31.74.195:56085)
0806 03:03:13.441850 (+ 970us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:13.441947 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":793,"thread_start_us":394,"threads_started":1}
I20260806 03:03:13.845444 15571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:13.844481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:33865 (local address 127.31.74.194:58745)
0806 03:03:13.845030 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:13.845135 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":304,"threads_started":1}
I20260806 03:03:14.245373 15572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:14.244340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:33865 (local address 127.31.74.193:34885)
0806 03:03:14.245062 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:14.245154 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":359,"threads_started":1}
I20260806 03:03:14.445346 15573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:14.444132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:33865 (local address 127.31.74.195:41929)
0806 03:03:14.444871 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:14.444976 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":622,"thread_start_us":423,"threads_started":1}
I20260806 03:03:14.544180 15574 raft_consensus.cc:482] T dbeeb9af3ace42c68c40c02a7ba33c3d P dc3142ac2c8548c48750d63e7eca5961 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:03:14.544541 15574 raft_consensus.cc:504] T dbeeb9af3ace42c68c40c02a7ba33c3d P dc3142ac2c8548c48750d63e7eca5961 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ee2abdf415246509365cee48c030d36" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 32815 } } peers { permanent_uuid: "dc3142ac2c8548c48750d63e7eca5961" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37831 } } peers { permanent_uuid: "bb22bd13bfd8404aa9d02498577cd212" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 45731 } }
I20260806 03:03:14.545365 15574 leader_election.cc:288] T dbeeb9af3ace42c68c40c02a7ba33c3d P dc3142ac2c8548c48750d63e7eca5961 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2ee2abdf415246509365cee48c030d36 (127.31.74.195:32815), bb22bd13bfd8404aa9d02498577cd212 (127.31.74.194:45731)
W20260806 03:03:14.546846 15149 leader_election.cc:341] T dbeeb9af3ace42c68c40c02a7ba33c3d P dc3142ac2c8548c48750d63e7eca5961 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bb22bd13bfd8404aa9d02498577cd212 (127.31.74.194:45731): Not found: Tablet not found: dbeeb9af3ace42c68c40c02a7ba33c3d
W20260806 03:03:14.547079 15147 leader_election.cc:341] T dbeeb9af3ace42c68c40c02a7ba33c3d P dc3142ac2c8548c48750d63e7eca5961 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2ee2abdf415246509365cee48c030d36 (127.31.74.195:32815): Not found: Tablet not found: dbeeb9af3ace42c68c40c02a7ba33c3d
I20260806 03:03:14.547178 15147 leader_election.cc:302] T dbeeb9af3ace42c68c40c02a7ba33c3d P dc3142ac2c8548c48750d63e7eca5961 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc3142ac2c8548c48750d63e7eca5961; no voters: 2ee2abdf415246509365cee48c030d36, bb22bd13bfd8404aa9d02498577cd212
I20260806 03:03:14.547902 15574 raft_consensus.cc:2726] T dbeeb9af3ace42c68c40c02a7ba33c3d P dc3142ac2c8548c48750d63e7eca5961 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:03:14.848161 15575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:14.846993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:33865 (local address 127.31.74.194:34485)
0806 03:03:14.847763 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:14.847921 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":379,"threads_started":1}
I20260806 03:03:15.247989 15576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:15.246849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:33865 (local address 127.31.74.193:46381)
0806 03:03:15.247647 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:15.247760 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":474,"threads_started":1}
I20260806 03:03:15.449056 15577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:15.447513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:33865 (local address 127.31.74.195:38839)
0806 03:03:15.448687 (+ 1174us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:15.448831 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1043,"thread_start_us":632,"threads_started":1}
I20260806 03:03:15.850486 15578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:15.849599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:33865 (local address 127.31.74.194:53033)
0806 03:03:15.850184 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:15.850283 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":345,"threads_started":1}
I20260806 03:03:16.250077 15579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:16.249313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:33865 (local address 127.31.74.193:57459)
0806 03:03:16.249782 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:16.249881 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":257,"threads_started":1}
I20260806 03:03:16.452190 15580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:16.450436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:33865 (local address 127.31.74.195:52967)
0806 03:03:16.451889 (+ 1453us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:16.451991 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1345,"thread_start_us":620,"threads_started":1}
I20260806 03:03:16.852842 15581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:16.851837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:33865 (local address 127.31.74.194:46561)
0806 03:03:16.852525 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:16.852623 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":352,"threads_started":1}
I20260806 03:03:17.252542 15582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:17.251557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:33865 (local address 127.31.74.193:51471)
0806 03:03:17.252238 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:17.252337 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":412,"threads_started":1}
I20260806 03:03:17.454317 15583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:17.453448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:33865 (local address 127.31.74.195:42309)
0806 03:03:17.454003 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:17.454105 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":278,"threads_started":1}
I20260806 03:03:17.854995 15584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:17.854096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:33865 (local address 127.31.74.194:54295)
0806 03:03:17.854707 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:17.854801 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":288,"threads_started":1}
I20260806 03:03:18.254937 15585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:18.253655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:33865 (local address 127.31.74.193:32871)
0806 03:03:18.254456 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:18.254692 (+ 236us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":431,"threads_started":1}
I20260806 03:03:18.456328 15586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:18.455477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:33865 (local address 127.31.74.195:32987)
0806 03:03:18.456034 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:18.456129 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":308,"threads_started":1}
I20260806 03:03:18.857224 15587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:18.855953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:33865 (local address 127.31.74.194:48977)
0806 03:03:18.856778 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:18.856897 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":356,"threads_started":1}
I20260806 03:03:19.257344 15588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:19.256486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:33865 (local address 127.31.74.193:58683)
0806 03:03:19.257015 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:19.257108 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":293,"threads_started":1}
I20260806 03:03:19.458380 15589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:19.457566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:33865 (local address 127.31.74.195:38381)
0806 03:03:19.458110 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:19.458211 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":286,"threads_started":1}
I20260806 03:03:19.859984 15590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:19.858877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:33865 (local address 127.31.74.194:56591)
0806 03:03:19.859620 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:19.859734 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":475,"threads_started":1}
I20260806 03:03:20.259832 15591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:20.258862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:33865 (local address 127.31.74.193:51297)
0806 03:03:20.259392 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:20.259500 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":259,"threads_started":1}
I20260806 03:03:20.460682 15592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:20.459846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:33865 (local address 127.31.74.195:38101)
0806 03:03:20.460383 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:20.460475 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":268,"threads_started":1}
I20260806 03:03:20.614675 15593 raft_consensus.cc:482] T dbeeb9af3ace42c68c40c02a7ba33c3d P dc3142ac2c8548c48750d63e7eca5961 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:03:20.614925 15593 raft_consensus.cc:504] T dbeeb9af3ace42c68c40c02a7ba33c3d P dc3142ac2c8548c48750d63e7eca5961 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ee2abdf415246509365cee48c030d36" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 32815 } } peers { permanent_uuid: "dc3142ac2c8548c48750d63e7eca5961" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 37831 } } peers { permanent_uuid: "bb22bd13bfd8404aa9d02498577cd212" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 45731 } }
I20260806 03:03:20.615753 15593 leader_election.cc:288] T dbeeb9af3ace42c68c40c02a7ba33c3d P dc3142ac2c8548c48750d63e7eca5961 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2ee2abdf415246509365cee48c030d36 (127.31.74.195:32815), bb22bd13bfd8404aa9d02498577cd212 (127.31.74.194:45731)
W20260806 03:03:20.617045 15147 leader_election.cc:341] T dbeeb9af3ace42c68c40c02a7ba33c3d P dc3142ac2c8548c48750d63e7eca5961 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2ee2abdf415246509365cee48c030d36 (127.31.74.195:32815): Not found: Tablet not found: dbeeb9af3ace42c68c40c02a7ba33c3d
W20260806 03:03:20.617348 15149 leader_election.cc:341] T dbeeb9af3ace42c68c40c02a7ba33c3d P dc3142ac2c8548c48750d63e7eca5961 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bb22bd13bfd8404aa9d02498577cd212 (127.31.74.194:45731): Not found: Tablet not found: dbeeb9af3ace42c68c40c02a7ba33c3d
I20260806 03:03:20.617528 15149 leader_election.cc:302] T dbeeb9af3ace42c68c40c02a7ba33c3d P dc3142ac2c8548c48750d63e7eca5961 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc3142ac2c8548c48750d63e7eca5961; no voters: 2ee2abdf415246509365cee48c030d36, bb22bd13bfd8404aa9d02498577cd212
I20260806 03:03:20.617841 15593 raft_consensus.cc:2726] T dbeeb9af3ace42c68c40c02a7ba33c3d P dc3142ac2c8548c48750d63e7eca5961 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:03:20.862890 15594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:20.861781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:33865 (local address 127.31.74.194:53903)
0806 03:03:20.862506 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:20.862704 (+ 198us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":431,"threads_started":1}
I20260806 03:03:21.262265 15595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:21.261260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:33865 (local address 127.31.74.193:50925)
0806 03:03:21.261993 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:21.262080 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":522,"threads_started":1}
I20260806 03:03:21.424188 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 15092
I20260806 03:03:21.450850 15596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:21.425976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55558 (local address 127.31.74.193:37831)
0806 03:03:21.426925 (+ 949us) server_negotiation.cc:206] Beginning negotiation
0806 03:03:21.426938 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 03:03:21.426980 (+ 42us) server_negotiation.cc:407] Connection header received
0806 03:03:21.427345 (+ 365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:03:21.427360 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:03:21.427685 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:03:21.427941 (+ 256us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:03:21.430397 (+ 2456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:21.431613 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:21.433388 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:21.434200 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:21.434323 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:03:21.443347 (+ 9024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:03:21.443366 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:03:21.443385 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:03:21.443438 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:03:21.447131 (+ 3693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:03:21.448281 (+ 1150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:03:21.448289 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:03:21.448292 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:03:21.448464 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:03:21.448946 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:03:21.448950 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:03:21.448952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:03:21.449458 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:03:21.449719 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0806 03:03:21.450154 (+ 435us) server_negotiation.cc:299] Negotiation successful
0806 03:03:21.450409 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":711,"thread_start_us":537,"threads_started":1}
I20260806 03:03:21.452548 15260 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 03:03:21.463639 15598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:21.462530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:33865 (local address 127.31.74.195:42445)
0806 03:03:21.463248 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:21.463392 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":369,"threads_started":1}
I20260806 03:03:21.561683 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 15092
I20260806 03:03:21.586002 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 15269
I20260806 03:03:21.605410 15599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:21.587297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40120 (local address 127.31.74.194:45731)
0806 03:03:21.587911 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0806 03:03:21.587916 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 03:03:21.587935 (+ 19us) server_negotiation.cc:407] Connection header received
0806 03:03:21.588000 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:03:21.588004 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:03:21.588070 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:03:21.588189 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:03:21.590297 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:21.591483 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:21.593576 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:21.594020 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:21.594140 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:03:21.597850 (+ 3710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:03:21.597858 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:03:21.597864 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:03:21.597914 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:03:21.601420 (+ 3506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:03:21.602392 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:03:21.602397 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:03:21.602399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:03:21.602548 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:03:21.603241 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:03:21.603248 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:03:21.603251 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:03:21.603947 (+ 696us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:03:21.604195 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0806 03:03:21.604608 (+ 413us) server_negotiation.cc:299] Negotiation successful
0806 03:03:21.604887 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":341,"threads_started":1}
I20260806 03:03:21.606319 15396 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 03:03:21.704276 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 15269
I20260806 03:03:21.726128 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 15405
I20260806 03:03:21.743667 15601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:21.727307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37588 (local address 127.31.74.195:32815)
0806 03:03:21.728071 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0806 03:03:21.728077 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 03:03:21.728102 (+ 25us) server_negotiation.cc:407] Connection header received
0806 03:03:21.728182 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:03:21.728186 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:03:21.728286 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:03:21.728411 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:03:21.730115 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:21.731334 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:21.732949 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:21.733468 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:21.733582 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:03:21.736825 (+ 3243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:03:21.736831 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:03:21.736835 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:03:21.736881 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:03:21.740236 (+ 3355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:03:21.741100 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:03:21.741107 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:03:21.741111 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:03:21.741315 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:03:21.741758 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:03:21.741765 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:03:21.741769 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:03:21.742116 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:03:21.742452 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0806 03:03:21.742941 (+ 489us) server_negotiation.cc:299] Negotiation successful
0806 03:03:21.743238 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":348,"threads_started":1}
I20260806 03:03:21.744560 15533 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 03:03:21.834625 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 15405
2026-08-06T03:03:21Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (27615 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 03:03:21 dist-test-slave-dhph krb5kdc[15605](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 03:03:21 dist-test-slave-dhph krb5kdc[15605](info): set up 2 sockets
Aug 06 03:03:21 dist-test-slave-dhph krb5kdc[15605](info): commencing operation
krb5kdc: starting...
W20260806 03:03:23.151962 32043 mini_kdc.cc:121] Time spent starting KDC: real 1.268s user 0.003s sys 0.057s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/krb5kdc/test-user.keytab.
Aug 06 03:03:23 dist-test-slave-dhph krb5kdc[15605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T03:03:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:03:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.74.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab.
Entry for principal kudu/127.31.74.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab.
I20260806 03:03:23.577627 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:39773
--webserver_interface=127.31.74.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab
--principal=kudu/127.31.74.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:40437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.74.254:39773
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:03:24.070323 15618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:03:24.071091 15618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:03:24.071583 15618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:03:24.085613 15618 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260806 03:03:24.085790 15618 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260806 03:03:24.085863 15618 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260806 03:03:24.085937 15618 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:03:24.086017 15618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:03:24.086127 15618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:03:24.086223 15618 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:03:24.086310 15618 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:03:24.109805 15618 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:40437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.31.74.254:39773
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab
--principal=kudu/127.31.74.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:39773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.74.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:03:24.111603 15618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:03:24.113920 15618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:03:24.132635 15624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:03:24.132879 15623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:03:24.132984 15626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:03:24.134449 15618 server_base.cc:1034] running on GCE node
I20260806 03:03:24.135789 15618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:03:24.137817 15618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:03:24.141389 15618 hybrid_clock.cc:648] HybridClock initialized: now 1785985404141151 us; error 42 us; skew 500 ppm
I20260806 03:03:24.141950 15618 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 03:03:24 dist-test-slave-dhph krb5kdc[15605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985404, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:03:24.146931 15618 init.cc:377] Logged in from keytab as kudu/127.31.74.254@KRBTEST.COM (short username kudu)
I20260806 03:03:24.149854 15618 webserver.cc:460] Webserver started at http://127.31.74.254:44417/ using document root <none> and password file <none>
I20260806 03:03:24.150815 15618 fs_manager.cc:359] Metadata directory not provided
I20260806 03:03:24.150969 15618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:03:24.151324 15618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:03:24.155717 15618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/master-0/data/instance:
uuid: "37607d7875bc405292c7a75a319e369a"
format_stamp: "Formatted at 2026-08-06 03:03:24 on dist-test-slave-dhph"
server_key: "9e0b3ff9e21e91e8b44ed055d16e0698"
server_key_iv: "d5562264c01f98fce9b14d48a61bf396"
server_key_version: "encryptionkey@0"
I20260806 03:03:24.156728 15618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/master-0/wal/instance:
uuid: "37607d7875bc405292c7a75a319e369a"
format_stamp: "Formatted at 2026-08-06 03:03:24 on dist-test-slave-dhph"
server_key: "9e0b3ff9e21e91e8b44ed055d16e0698"
server_key_iv: "d5562264c01f98fce9b14d48a61bf396"
server_key_version: "encryptionkey@0"
I20260806 03:03:24.164767 15618 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.001s
I20260806 03:03:24.170075 15633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:03:24.173266 15618 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260806 03:03:24.173547 15618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/master-0/wal
uuid: "37607d7875bc405292c7a75a319e369a"
format_stamp: "Formatted at 2026-08-06 03:03:24 on dist-test-slave-dhph"
server_key: "9e0b3ff9e21e91e8b44ed055d16e0698"
server_key_iv: "d5562264c01f98fce9b14d48a61bf396"
server_key_version: "encryptionkey@0"
I20260806 03:03:24.174049 15618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:03:24.213493 15618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:03:24.220515 15618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:03:24.221071 15618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:03:24.246328 15618 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.254:39773
I20260806 03:03:24.246364 15689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.254:39773 every 8 connection(s)
I20260806 03:03:24.248385 15618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
I20260806 03:03:24.253021 15690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:03:24.255442 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 15618
I20260806 03:03:24.255750 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/master-0/wal/instance
I20260806 03:03:24.256472 32043 external_mini_cluster.cc:1428] Setting key b42115d3c834bbc29e64fa7ffb442cb2
I20260806 03:03:24.275583 15690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37607d7875bc405292c7a75a319e369a: Bootstrap starting.
Aug 06 03:03:24 dist-test-slave-dhph krb5kdc[15605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 03:03:24.279954 15690 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 37607d7875bc405292c7a75a319e369a: Neither blocks nor log segments found. Creating new log.
I20260806 03:03:24.282162 15690 log.cc:824] T 00000000000000000000000000000000 P 37607d7875bc405292c7a75a319e369a: Log is configured to *not* fsync() on all Append() calls
I20260806 03:03:24.287285 15690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37607d7875bc405292c7a75a319e369a: No bootstrap required, opened a new log
I20260806 03:03:24.290400 15693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:24.259265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56236 (local address 127.31.74.254:39773)
0806 03:03:24.260553 (+ 1288us) server_negotiation.cc:206] Beginning negotiation
0806 03:03:24.260578 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0806 03:03:24.260645 (+ 67us) server_negotiation.cc:407] Connection header received
0806 03:03:24.263065 (+ 2420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:03:24.263104 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:03:24.263794 (+ 690us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:03:24.264311 (+ 517us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:03:24.266077 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:24.268721 (+ 2644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:24.271013 (+ 2292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:24.273903 (+ 2890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:24.274181 (+ 278us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:03:24.279747 (+ 5566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:03:24.279764 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:03:24.279792 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:03:24.279862 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:03:24.284601 (+ 4739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:03:24.285596 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:03:24.285603 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:03:24.285610 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:03:24.285808 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:03:24.286307 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:03:24.286311 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:03:24.286314 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:03:24.286971 (+ 657us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:03:24.287291 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0806 03:03:24.287898 (+ 607us) server_negotiation.cc:299] Negotiation successful
0806 03:03:24.288394 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":984,"thread_start_us":650,"threads_started":1}
I20260806 03:03:24.304371 15690 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37607d7875bc405292c7a75a319e369a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37607d7875bc405292c7a75a319e369a" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 39773 } }
I20260806 03:03:24.305056 15690 raft_consensus.cc:374] T 00000000000000000000000000000000 P 37607d7875bc405292c7a75a319e369a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:03:24.305306 15690 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37607d7875bc405292c7a75a319e369a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37607d7875bc405292c7a75a319e369a, State: Initialized, Role: FOLLOWER
I20260806 03:03:24.306140 15690 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37607d7875bc405292c7a75a319e369a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37607d7875bc405292c7a75a319e369a" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 39773 } }
I20260806 03:03:24.306420 15690 raft_consensus.cc:388] T 00000000000000000000000000000000 P 37607d7875bc405292c7a75a319e369a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:03:24.306546 15690 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37607d7875bc405292c7a75a319e369a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:03:24.306780 15690 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37607d7875bc405292c7a75a319e369a [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:03:24.309423 15690 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37607d7875bc405292c7a75a319e369a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37607d7875bc405292c7a75a319e369a" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 39773 } }
I20260806 03:03:24.310909 15690 leader_election.cc:302] T 00000000000000000000000000000000 P 37607d7875bc405292c7a75a319e369a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37607d7875bc405292c7a75a319e369a; no voters:
I20260806 03:03:24.312600 15695 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37607d7875bc405292c7a75a319e369a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:03:24.312772 15690 leader_election.cc:288] T 00000000000000000000000000000000 P 37607d7875bc405292c7a75a319e369a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:03:24.313737 15695 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37607d7875bc405292c7a75a319e369a [term 1 LEADER]: Becoming Leader. State: Replica: 37607d7875bc405292c7a75a319e369a, State: Running, Role: LEADER
I20260806 03:03:24.315027 15695 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37607d7875bc405292c7a75a319e369a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37607d7875bc405292c7a75a319e369a" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 39773 } }
I20260806 03:03:24.320293 15690 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37607d7875bc405292c7a75a319e369a [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:03:24.320465 15696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37607d7875bc405292c7a75a319e369a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37607d7875bc405292c7a75a319e369a. Latest consensus state: current_term: 1 leader_uuid: "37607d7875bc405292c7a75a319e369a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37607d7875bc405292c7a75a319e369a" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 39773 } } }
I20260806 03:03:24.320938 15696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37607d7875bc405292c7a75a319e369a [sys.catalog]: This master's current role is: LEADER
I20260806 03:03:24.321508 15697 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37607d7875bc405292c7a75a319e369a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "37607d7875bc405292c7a75a319e369a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37607d7875bc405292c7a75a319e369a" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 39773 } } }
I20260806 03:03:24.321908 15697 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37607d7875bc405292c7a75a319e369a [sys.catalog]: This master's current role is: LEADER
I20260806 03:03:24.323997 15699 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:03:24.331993 15699 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:03:24.348753 15699 catalog_manager.cc:1347] Generated new cluster ID: be8c261140b64d3db0b57e646b399eca
I20260806 03:03:24.349473 15699 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:03:24.368903 15699 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:03:24.371174 15699 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:03:24.399869 15699 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 37607d7875bc405292c7a75a319e369a: Generated new TSK 0
I20260806 03:03:24.402104 15699 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.74.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.193.keytab.
Entry for principal kudu/127.31.74.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.193.keytab.
I20260806 03:03:24.512811 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.193:0
--local_ip_for_outbound_sockets=127.31.74.193
--webserver_interface=127.31.74.193
--webserver_port=0
--tserver_master_addrs=127.31.74.254:39773
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.193.keytab
--principal=kudu/127.31.74.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:40437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:03:24.974653 15716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:03:24.975066 15716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:03:24.975180 15716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:03:24.985729 15716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:03:24.985875 15716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:03:24.986002 15716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.193
I20260806 03:03:24.997975 15716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:40437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.193.keytab
--principal=kudu/127.31.74.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.74.193
--webserver_port=0
--tserver_master_addrs=127.31.74.254:39773
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.193
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:03:25.000018 15716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:03:25.001600 15716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:03:25.020378 15721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:03:25.021140 15724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:03:25.020385 15722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:03:25.024173 15716 server_base.cc:1034] running on GCE node
I20260806 03:03:25.025010 15716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:03:25.026223 15716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:03:25.027626 15716 hybrid_clock.cc:648] HybridClock initialized: now 1785985405027543 us; error 67 us; skew 500 ppm
I20260806 03:03:25.028106 15716 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 03:03:25 dist-test-slave-dhph krb5kdc[15605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985405, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:03:25.032018 15716 init.cc:377] Logged in from keytab as kudu/127.31.74.193@KRBTEST.COM (short username kudu)
I20260806 03:03:25.033967 15716 webserver.cc:460] Webserver started at http://127.31.74.193:43289/ using document root <none> and password file <none>
I20260806 03:03:25.035116 15716 fs_manager.cc:359] Metadata directory not provided
I20260806 03:03:25.035372 15716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:03:25.035717 15716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:03:25.040355 15716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-0/data/instance:
uuid: "9ee24a099afe42d89e3caeb7d9d298a4"
format_stamp: "Formatted at 2026-08-06 03:03:25 on dist-test-slave-dhph"
server_key: "0a2a669a625632fc1eeeb020390b2e85"
server_key_iv: "0a40ee273a22047da0adebd449b0326d"
server_key_version: "encryptionkey@0"
I20260806 03:03:25.042289 15716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-0/wal/instance:
uuid: "9ee24a099afe42d89e3caeb7d9d298a4"
format_stamp: "Formatted at 2026-08-06 03:03:25 on dist-test-slave-dhph"
server_key: "0a2a669a625632fc1eeeb020390b2e85"
server_key_iv: "0a40ee273a22047da0adebd449b0326d"
server_key_version: "encryptionkey@0"
I20260806 03:03:25.050963 15716 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.002s sys 0.005s
I20260806 03:03:25.056175 15731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:03:25.058355 15716 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260806 03:03:25.058569 15716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-0/wal
uuid: "9ee24a099afe42d89e3caeb7d9d298a4"
format_stamp: "Formatted at 2026-08-06 03:03:25 on dist-test-slave-dhph"
server_key: "0a2a669a625632fc1eeeb020390b2e85"
server_key_iv: "0a40ee273a22047da0adebd449b0326d"
server_key_version: "encryptionkey@0"
I20260806 03:03:25.058895 15716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:03:25.078097 15716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:03:25.084173 15716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:03:25.084457 15716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:03:25.086272 15716 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:03:25.088439 15716 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:03:25.088541 15716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:03:25.088647 15716 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:03:25.088702 15716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:03:25.136627 15716 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.193:38719
I20260806 03:03:25.136685 15848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.193:38719 every 8 connection(s)
I20260806 03:03:25.138281 15716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
I20260806 03:03:25.148038 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 15716
I20260806 03:03:25.148397 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-0/wal/instance
I20260806 03:03:25.149304 32043 external_mini_cluster.cc:1428] Setting key 20004cb0487c18d634c49a0a132104af
Aug 06 03:03:25 dist-test-slave-dhph krb5kdc[15605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985405, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.193@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 03:03:25.168756 15851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:25.141495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.193:45247 (local address 127.31.74.254:39773)
0806 03:03:25.142023 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0806 03:03:25.142029 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 03:03:25.143467 (+ 1438us) server_negotiation.cc:407] Connection header received
0806 03:03:25.145346 (+ 1879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:03:25.145351 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:03:25.145431 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:03:25.145543 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:03:25.147426 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:25.148840 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:25.151969 (+ 3129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:25.152522 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:25.152638 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:03:25.159046 (+ 6408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:03:25.159057 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:03:25.159064 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:03:25.159125 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:03:25.163301 (+ 4176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:03:25.164710 (+ 1409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:03:25.164719 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:03:25.164725 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:03:25.164905 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:03:25.165525 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:03:25.165533 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:03:25.165539 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:03:25.165971 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:03:25.166149 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0806 03:03:25.167591 (+ 1442us) server_negotiation.cc:299] Negotiation successful
0806 03:03:25.167908 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":251,"threads_started":1}
I20260806 03:03:25.169853 15852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:25.142011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:39773 (local address 127.31.74.193:45247)
0806 03:03:25.143237 (+ 1226us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:25.143282 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 03:03:25.145029 (+ 1747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:03:25.145820 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:03:25.145829 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:03:25.146421 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:03:25.147145 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:03:25.147171 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:03:25.149084 (+ 1913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:03:25.149090 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:03:25.151677 (+ 2587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:03:25.151704 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:03:25.152828 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:03:25.152838 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:03:25.153207 (+ 369us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:03:25.154254 (+ 1047us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:03:25.154324 (+ 70us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:03:25.158684 (+ 4360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:03:25.163543 (+ 4859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:03:25.163554 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:03:25.163571 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:03:25.164533 (+ 962us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:03:25.165057 (+ 524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:03:25.165061 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:03:25.165064 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:03:25.165383 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:03:25.166249 (+ 866us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:03:25.166259 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:03:25.167032 (+ 773us) client_negotiation.cc:769] Sending connection context
0806 03:03:25.167426 (+ 394us) client_negotiation.cc:240] Negotiation successful
0806 03:03:25.168065 (+ 639us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":324,"threads_started":1}
I20260806 03:03:25.172821 15849 heartbeater.cc:347] Connected to a master server at 127.31.74.254:39773
I20260806 03:03:25.173211 15849 heartbeater.cc:464] Registering TS with master...
I20260806 03:03:25.174438 15849 heartbeater.cc:511] Master 127.31.74.254:39773 requested a full tablet report, sending...
I20260806 03:03:25.177542 15653 ts_manager.cc:194] Registered new tserver with Master: 9ee24a099afe42d89e3caeb7d9d298a4 (127.31.74.193:38719)
I20260806 03:03:25.179587 15653 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.74.193@KRBTEST.COM'} at 127.31.74.193:45247
WARNING: no policy specified for kudu/127.31.74.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.194.keytab.
Entry for principal kudu/127.31.74.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.194.keytab.
I20260806 03:03:25.244915 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.194:0
--local_ip_for_outbound_sockets=127.31.74.194
--webserver_interface=127.31.74.194
--webserver_port=0
--tserver_master_addrs=127.31.74.254:39773
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.194.keytab
--principal=kudu/127.31.74.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:40437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:03:25.661559 15855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:03:25.661954 15855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:03:25.662146 15855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:03:25.671499 15855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:03:25.671700 15855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:03:25.671890 15855 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.194
I20260806 03:03:25.684855 15855 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:40437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.194.keytab
--principal=kudu/127.31.74.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.74.194
--webserver_port=0
--tserver_master_addrs=127.31.74.254:39773
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.194
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:03:25.686439 15855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:03:25.688024 15855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:03:25.706712 15860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:03:25.706800 15864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:03:25.706665 15861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:03:25.708130 15855 server_base.cc:1034] running on GCE node
I20260806 03:03:25.709004 15855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:03:25.710107 15855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:03:25.711377 15855 hybrid_clock.cc:648] HybridClock initialized: now 1785985405711335 us; error 53 us; skew 500 ppm
I20260806 03:03:25.711872 15855 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 03:03:25 dist-test-slave-dhph krb5kdc[15605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985405, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:03:25.716506 15855 init.cc:377] Logged in from keytab as kudu/127.31.74.194@KRBTEST.COM (short username kudu)
I20260806 03:03:25.718446 15855 webserver.cc:460] Webserver started at http://127.31.74.194:40765/ using document root <none> and password file <none>
I20260806 03:03:25.719275 15855 fs_manager.cc:359] Metadata directory not provided
I20260806 03:03:25.719431 15855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:03:25.719770 15855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:03:25.723148 15855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-1/data/instance:
uuid: "14d8bb5a6aeb49c99ded6a54400fae01"
format_stamp: "Formatted at 2026-08-06 03:03:25 on dist-test-slave-dhph"
server_key: "1b2eaf3a507f0a1fcf401be9467ab5b3"
server_key_iv: "adb89725606df4b004c1ffcd0989a03e"
server_key_version: "encryptionkey@0"
I20260806 03:03:25.724014 15855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-1/wal/instance:
uuid: "14d8bb5a6aeb49c99ded6a54400fae01"
format_stamp: "Formatted at 2026-08-06 03:03:25 on dist-test-slave-dhph"
server_key: "1b2eaf3a507f0a1fcf401be9467ab5b3"
server_key_iv: "adb89725606df4b004c1ffcd0989a03e"
server_key_version: "encryptionkey@0"
I20260806 03:03:25.732883 15855 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.009s sys 0.000s
I20260806 03:03:25.737867 15871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:03:25.740309 15855 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260806 03:03:25.740558 15855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-1/wal
uuid: "14d8bb5a6aeb49c99ded6a54400fae01"
format_stamp: "Formatted at 2026-08-06 03:03:25 on dist-test-slave-dhph"
server_key: "1b2eaf3a507f0a1fcf401be9467ab5b3"
server_key_iv: "adb89725606df4b004c1ffcd0989a03e"
server_key_version: "encryptionkey@0"
I20260806 03:03:25.740809 15855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:03:25.774513 15855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:03:25.779937 15855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:03:25.780368 15855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:03:25.781644 15855 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:03:25.783377 15855 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:03:25.783488 15855 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:03:25.783646 15855 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:03:25.783730 15855 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:03:25.832826 15855 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.194:35369
I20260806 03:03:25.833119 15989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.194:35369 every 8 connection(s)
I20260806 03:03:25.834401 15855 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
I20260806 03:03:25.835173 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 15855
I20260806 03:03:25.835423 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-1/wal/instance
I20260806 03:03:25.836491 32043 external_mini_cluster.cc:1428] Setting key 310485107a552035e56a31c36c509f99
Aug 06 03:03:25 dist-test-slave-dhph krb5kdc[15605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985405, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.194@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 03:03:25.868772 15992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:25.838074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.194:32771 (local address 127.31.74.254:39773)
0806 03:03:25.838659 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0806 03:03:25.838666 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 03:03:25.840781 (+ 2115us) server_negotiation.cc:407] Connection header received
0806 03:03:25.843209 (+ 2428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:03:25.843283 (+ 74us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:03:25.843403 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:03:25.844038 (+ 635us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:03:25.848768 (+ 4730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:25.849893 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:25.852849 (+ 2956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:25.853440 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:25.853608 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:03:25.859031 (+ 5423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:03:25.859047 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:03:25.859056 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:03:25.859136 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:03:25.863598 (+ 4462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:03:25.865004 (+ 1406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:03:25.865015 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:03:25.865022 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:03:25.865264 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:03:25.865908 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:03:25.865917 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:03:25.865924 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:03:25.866490 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:03:25.866752 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0806 03:03:25.867857 (+ 1105us) server_negotiation.cc:299] Negotiation successful
0806 03:03:25.868202 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":304,"threads_started":1}
I20260806 03:03:25.869514 15993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:25.838910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:39773 (local address 127.31.74.194:32771)
0806 03:03:25.840235 (+ 1325us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:25.840297 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0806 03:03:25.842523 (+ 2226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:03:25.845633 (+ 3110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:03:25.845646 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:03:25.846939 (+ 1293us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:03:25.848405 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:03:25.848440 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:03:25.850235 (+ 1795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:03:25.850242 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:03:25.852504 (+ 2262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:03:25.852528 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:03:25.853745 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:03:25.853754 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:03:25.854066 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:03:25.854968 (+ 902us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:03:25.854996 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:03:25.858558 (+ 3562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:03:25.864209 (+ 5651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:03:25.864219 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:03:25.864237 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:03:25.864817 (+ 580us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:03:25.865450 (+ 633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:03:25.865455 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:03:25.865458 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:03:25.865700 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:03:25.866883 (+ 1183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:03:25.866891 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:03:25.867369 (+ 478us) client_negotiation.cc:769] Sending connection context
0806 03:03:25.867650 (+ 281us) client_negotiation.cc:240] Negotiation successful
0806 03:03:25.868153 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1013,"thread_start_us":696,"threads_started":1}
I20260806 03:03:25.872109 15990 heartbeater.cc:347] Connected to a master server at 127.31.74.254:39773
I20260806 03:03:25.872630 15990 heartbeater.cc:464] Registering TS with master...
I20260806 03:03:25.873733 15990 heartbeater.cc:511] Master 127.31.74.254:39773 requested a full tablet report, sending...
I20260806 03:03:25.876327 15653 ts_manager.cc:194] Registered new tserver with Master: 14d8bb5a6aeb49c99ded6a54400fae01 (127.31.74.194:35369)
I20260806 03:03:25.877321 15653 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.74.194@KRBTEST.COM'} at 127.31.74.194:32771
WARNING: no policy specified for kudu/127.31.74.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.195.keytab.
Entry for principal kudu/127.31.74.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.195.keytab.
W20260806 03:03:25.941841 32043 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.31.74.195: real 0.101s user 0.001s sys 0.031s
I20260806 03:03:25.943147 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.195:0
--local_ip_for_outbound_sockets=127.31.74.195
--webserver_interface=127.31.74.195
--webserver_port=0
--tserver_master_addrs=127.31.74.254:39773
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.195.keytab
--principal=kudu/127.31.74.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:40437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 03:03:26.186893 15849 heartbeater.cc:503] Master 127.31.74.254:39773 was elected leader, sending a full tablet report...
W20260806 03:03:26.414961 15996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:03:26.415948 15996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:03:26.416250 15996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:03:26.427651 15996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:03:26.427829 15996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:03:26.428056 15996 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.195
I20260806 03:03:26.440376 15996 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:40437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.195.keytab
--principal=kudu/127.31.74.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.74.195
--webserver_port=0
--tserver_master_addrs=127.31.74.254:39773
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.195
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:03:26.442415 15996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:03:26.443925 15996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:03:26.461091 16001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:03:26.461303 16002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:03:26.461421 16004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:03:26.463554 15996 server_base.cc:1034] running on GCE node
I20260806 03:03:26.464745 15996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:03:26.466209 15996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:03:26.467555 15996 hybrid_clock.cc:648] HybridClock initialized: now 1785985406467462 us; error 90 us; skew 500 ppm
I20260806 03:03:26.468096 15996 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 03:03:26 dist-test-slave-dhph krb5kdc[15605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985406, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:03:26.476572 15996 init.cc:377] Logged in from keytab as kudu/127.31.74.195@KRBTEST.COM (short username kudu)
I20260806 03:03:26.478768 15996 webserver.cc:460] Webserver started at http://127.31.74.195:46653/ using document root <none> and password file <none>
I20260806 03:03:26.479710 15996 fs_manager.cc:359] Metadata directory not provided
I20260806 03:03:26.479894 15996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:03:26.480300 15996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:03:26.484234 15996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-2/data/instance:
uuid: "50ddcdd9d41e4c0ea5d0eeae32f6e187"
format_stamp: "Formatted at 2026-08-06 03:03:26 on dist-test-slave-dhph"
server_key: "662ead0260e8c6e2922b512030cc50af"
server_key_iv: "99dccf488dd9e5728c50391ffc23c98c"
server_key_version: "encryptionkey@0"
I20260806 03:03:26.485152 15996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-2/wal/instance:
uuid: "50ddcdd9d41e4c0ea5d0eeae32f6e187"
format_stamp: "Formatted at 2026-08-06 03:03:26 on dist-test-slave-dhph"
server_key: "662ead0260e8c6e2922b512030cc50af"
server_key_iv: "99dccf488dd9e5728c50391ffc23c98c"
server_key_version: "encryptionkey@0"
I20260806 03:03:26.492216 15996 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260806 03:03:26.496882 16011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:03:26.499105 15996 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260806 03:03:26.499388 15996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-2/wal
uuid: "50ddcdd9d41e4c0ea5d0eeae32f6e187"
format_stamp: "Formatted at 2026-08-06 03:03:26 on dist-test-slave-dhph"
server_key: "662ead0260e8c6e2922b512030cc50af"
server_key_iv: "99dccf488dd9e5728c50391ffc23c98c"
server_key_version: "encryptionkey@0"
I20260806 03:03:26.499706 15996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:03:26.522149 15996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:03:26.528586 15996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:03:26.528944 15996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:03:26.530524 15996 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:03:26.532167 15996 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:03:26.532284 15996 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:03:26.532454 15996 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:03:26.532529 15996 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:03:26.578539 15996 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.195:32799
I20260806 03:03:26.578747 16125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.195:32799 every 8 connection(s)
I20260806 03:03:26.580051 15996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
I20260806 03:03:26.589395 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 15996
I20260806 03:03:26.589636 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-2/wal/instance
I20260806 03:03:26.590127 32043 external_mini_cluster.cc:1428] Setting key 4c0487284ac2ecc8b8017b0a1ae67a85
Aug 06 03:03:26 dist-test-slave-dhph krb5kdc[15605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985406, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.195@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 03:03:26.608651 16129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:26.584124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:39773 (local address 127.31.74.195:46627)
0806 03:03:26.585010 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:26.585052 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 03:03:26.586953 (+ 1901us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:03:26.587868 (+ 915us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:03:26.587877 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:03:26.588536 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:03:26.589611 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:03:26.589646 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:03:26.591023 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:03:26.591035 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:03:26.592908 (+ 1873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:03:26.592923 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:03:26.593788 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:03:26.593796 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:03:26.594040 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:03:26.595061 (+ 1021us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:03:26.595088 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:03:26.597831 (+ 2743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:03:26.603304 (+ 5473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:03:26.603318 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:03:26.603341 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:03:26.604149 (+ 808us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:03:26.604656 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:03:26.604661 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:03:26.604664 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:03:26.604971 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:03:26.605830 (+ 859us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:03:26.605838 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:03:26.606354 (+ 516us) client_negotiation.cc:769] Sending connection context
0806 03:03:26.606739 (+ 385us) client_negotiation.cc:240] Negotiation successful
0806 03:03:26.607343 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":324,"threads_started":1}
I20260806 03:03:26.609388 16128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:26.583534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.195:46627 (local address 127.31.74.254:39773)
0806 03:03:26.584103 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0806 03:03:26.584110 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 03:03:26.585199 (+ 1089us) server_negotiation.cc:407] Connection header received
0806 03:03:26.587293 (+ 2094us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:03:26.587304 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:03:26.587408 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:03:26.587516 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:03:26.589897 (+ 2381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:26.590849 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:26.593091 (+ 2242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:26.593531 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:26.593682 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:03:26.598201 (+ 4519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:03:26.598212 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:03:26.598221 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:03:26.598280 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:03:26.603089 (+ 4809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:03:26.604328 (+ 1239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:03:26.604335 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:03:26.604338 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:03:26.604503 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:03:26.605142 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:03:26.605147 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:03:26.605150 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:03:26.605584 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:03:26.605735 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0806 03:03:26.607253 (+ 1518us) server_negotiation.cc:299] Negotiation successful
0806 03:03:26.608177 (+ 924us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":291,"threads_started":1}
I20260806 03:03:26.612708 16126 heartbeater.cc:347] Connected to a master server at 127.31.74.254:39773
I20260806 03:03:26.614244 16126 heartbeater.cc:464] Registering TS with master...
I20260806 03:03:26.615262 16126 heartbeater.cc:511] Master 127.31.74.254:39773 requested a full tablet report, sending...
I20260806 03:03:26.617467 15653 ts_manager.cc:194] Registered new tserver with Master: 50ddcdd9d41e4c0ea5d0eeae32f6e187 (127.31.74.195:32799)
I20260806 03:03:26.618924 15653 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.74.195@KRBTEST.COM'} at 127.31.74.195:46627
I20260806 03:03:26.621474 32043 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 03:03:26.651271 16128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:26.632147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56252 (local address 127.31.74.254:39773)
0806 03:03:26.632441 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0806 03:03:26.632451 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 03:03:26.633071 (+ 620us) server_negotiation.cc:407] Connection header received
0806 03:03:26.633333 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:03:26.633338 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:03:26.633446 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:03:26.633546 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:03:26.635457 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:26.636404 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:26.638816 (+ 2412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:26.639429 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:26.639576 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:03:26.643495 (+ 3919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:03:26.643502 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:03:26.643505 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:03:26.643568 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:03:26.647293 (+ 3725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:03:26.648343 (+ 1050us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:03:26.648350 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:03:26.648353 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:03:26.648738 (+ 385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:03:26.649226 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:03:26.649231 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:03:26.649233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:03:26.649641 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:03:26.649858 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0806 03:03:26.650679 (+ 821us) server_negotiation.cc:299] Negotiation successful
0806 03:03:26.650905 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260806 03:03:26.659538 15653 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56252:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 03:03:26.668048 15653 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 03:03:26.699190 16139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:26.683644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:38719 (local address 127.0.0.1:55010)
0806 03:03:26.685424 (+ 1780us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:26.685498 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0806 03:03:26.685756 (+ 258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:03:26.688943 (+ 3187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:03:26.688952 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:03:26.689001 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:03:26.689920 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:03:26.689932 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:03:26.692845 (+ 2913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:03:26.692850 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:03:26.695532 (+ 2682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:03:26.695552 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:03:26.698105 (+ 2553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:03:26.698114 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:03:26.698331 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:03:26.698369 (+ 38us) client_negotiation.cc:769] Sending connection context
0806 03:03:26.698536 (+ 167us) client_negotiation.cc:240] Negotiation successful
0806 03:03:26.698794 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1100,"thread_start_us":624,"threads_started":1}
I20260806 03:03:26.699180 16140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:26.683833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55010 (local address 127.31.74.193:38719)
0806 03:03:26.686708 (+ 2875us) server_negotiation.cc:206] Beginning negotiation
0806 03:03:26.686756 (+ 48us) server_negotiation.cc:399] Waiting for connection header
0806 03:03:26.686895 (+ 139us) server_negotiation.cc:407] Connection header received
0806 03:03:26.687021 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:03:26.687030 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:03:26.687498 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:03:26.690176 (+ 2678us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:03:26.690513 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:26.692590 (+ 2077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:26.695776 (+ 3186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:26.697750 (+ 1974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:26.697901 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:03:26.698164 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0806 03:03:26.698624 (+ 460us) server_negotiation.cc:299] Negotiation successful
0806 03:03:26.698803 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2531,"thread_start_us":2324,"threads_started":1}
I20260806 03:03:26.701912 16141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:26.683976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.194:35369 (local address 127.0.0.1:56862)
0806 03:03:26.687149 (+ 3173us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:26.687173 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 03:03:26.687406 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:03:26.689896 (+ 2490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:03:26.689900 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:03:26.689929 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:03:26.690453 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:03:26.690468 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:03:26.692793 (+ 2325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:03:26.692801 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:03:26.697918 (+ 5117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:03:26.697942 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:03:26.701040 (+ 3098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:03:26.701046 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:03:26.701239 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:03:26.701282 (+ 43us) client_negotiation.cc:769] Sending connection context
0806 03:03:26.701390 (+ 108us) client_negotiation.cc:240] Negotiation successful
0806 03:03:26.701539 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3049,"thread_start_us":2620,"threads_started":1}
I20260806 03:03:26.701959 16142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:26.684434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56862 (local address 127.31.74.194:35369)
0806 03:03:26.686328 (+ 1894us) server_negotiation.cc:206] Beginning negotiation
0806 03:03:26.686356 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0806 03:03:26.687640 (+ 1284us) server_negotiation.cc:407] Connection header received
0806 03:03:26.688930 (+ 1290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:03:26.688941 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:03:26.689276 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:03:26.689607 (+ 331us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:03:26.690789 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:26.692589 (+ 1800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:26.698229 (+ 5640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:26.700807 (+ 2578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:26.700982 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:03:26.701192 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0806 03:03:26.701454 (+ 262us) server_negotiation.cc:299] Negotiation successful
0806 03:03:26.701632 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1604,"thread_start_us":1409,"threads_started":1}
I20260806 03:03:26.705452 15782 tablet_service.cc:1467] Processing CreateTablet for tablet 34c4e61c8a384b5696e55a0dd5b77d4e (DEFAULT_TABLE table=test-table [id=f79950d8da1e43ef979eab10a3abe8db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:03:26.705755 15921 tablet_service.cc:1467] Processing CreateTablet for tablet 34c4e61c8a384b5696e55a0dd5b77d4e (DEFAULT_TABLE table=test-table [id=f79950d8da1e43ef979eab10a3abe8db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:03:26.707134 15921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34c4e61c8a384b5696e55a0dd5b77d4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:03:26.707151 15782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34c4e61c8a384b5696e55a0dd5b77d4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:03:26.708103 16143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:26.687147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.195:32799 (local address 127.0.0.1:49924)
0806 03:03:26.696519 (+ 9372us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:26.696562 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 03:03:26.696719 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:03:26.697702 (+ 983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:03:26.697711 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:03:26.697748 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:03:26.698321 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:03:26.698445 (+ 124us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:03:26.702707 (+ 4262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:03:26.702712 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:03:26.704997 (+ 2285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:03:26.705017 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:03:26.707087 (+ 2070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:03:26.707094 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:03:26.707417 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:03:26.707459 (+ 42us) client_negotiation.cc:769] Sending connection context
0806 03:03:26.707571 (+ 112us) client_negotiation.cc:240] Negotiation successful
0806 03:03:26.707723 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9214,"thread_start_us":9008,"threads_started":1}
I20260806 03:03:26.709484 16144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:26.687458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49924 (local address 127.31.74.195:32799)
0806 03:03:26.693822 (+ 6364us) server_negotiation.cc:206] Beginning negotiation
0806 03:03:26.693832 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 03:03:26.696991 (+ 3159us) server_negotiation.cc:407] Connection header received
0806 03:03:26.697116 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:03:26.697126 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:03:26.697444 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:03:26.697683 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:03:26.698739 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:26.702493 (+ 3754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:26.705201 (+ 2708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:26.706826 (+ 1625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:26.706980 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:03:26.707238 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0806 03:03:26.708916 (+ 1678us) server_negotiation.cc:299] Negotiation successful
0806 03:03:26.709130 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6120,"thread_start_us":5925,"threads_started":1}
I20260806 03:03:26.716197 16060 tablet_service.cc:1467] Processing CreateTablet for tablet 34c4e61c8a384b5696e55a0dd5b77d4e (DEFAULT_TABLE table=test-table [id=f79950d8da1e43ef979eab10a3abe8db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:03:26.718181 16060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34c4e61c8a384b5696e55a0dd5b77d4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:03:26.724900 16145 tablet_bootstrap.cc:492] T 34c4e61c8a384b5696e55a0dd5b77d4e P 9ee24a099afe42d89e3caeb7d9d298a4: Bootstrap starting.
I20260806 03:03:26.727311 16146 tablet_bootstrap.cc:492] T 34c4e61c8a384b5696e55a0dd5b77d4e P 14d8bb5a6aeb49c99ded6a54400fae01: Bootstrap starting.
I20260806 03:03:26.730623 16145 tablet_bootstrap.cc:654] T 34c4e61c8a384b5696e55a0dd5b77d4e P 9ee24a099afe42d89e3caeb7d9d298a4: Neither blocks nor log segments found. Creating new log.
I20260806 03:03:26.730973 16146 tablet_bootstrap.cc:654] T 34c4e61c8a384b5696e55a0dd5b77d4e P 14d8bb5a6aeb49c99ded6a54400fae01: Neither blocks nor log segments found. Creating new log.
I20260806 03:03:26.732323 16145 log.cc:824] T 34c4e61c8a384b5696e55a0dd5b77d4e P 9ee24a099afe42d89e3caeb7d9d298a4: Log is configured to *not* fsync() on all Append() calls
I20260806 03:03:26.732324 16146 log.cc:824] T 34c4e61c8a384b5696e55a0dd5b77d4e P 14d8bb5a6aeb49c99ded6a54400fae01: Log is configured to *not* fsync() on all Append() calls
I20260806 03:03:26.736074 16145 tablet_bootstrap.cc:492] T 34c4e61c8a384b5696e55a0dd5b77d4e P 9ee24a099afe42d89e3caeb7d9d298a4: No bootstrap required, opened a new log
I20260806 03:03:26.736505 16145 ts_tablet_manager.cc:1397] T 34c4e61c8a384b5696e55a0dd5b77d4e P 9ee24a099afe42d89e3caeb7d9d298a4: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.007s
I20260806 03:03:26.737535 16146 tablet_bootstrap.cc:492] T 34c4e61c8a384b5696e55a0dd5b77d4e P 14d8bb5a6aeb49c99ded6a54400fae01: No bootstrap required, opened a new log
I20260806 03:03:26.738170 16146 ts_tablet_manager.cc:1397] T 34c4e61c8a384b5696e55a0dd5b77d4e P 14d8bb5a6aeb49c99ded6a54400fae01: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.003s
I20260806 03:03:26.739289 16149 tablet_bootstrap.cc:492] T 34c4e61c8a384b5696e55a0dd5b77d4e P 50ddcdd9d41e4c0ea5d0eeae32f6e187: Bootstrap starting.
I20260806 03:03:26.744412 16149 tablet_bootstrap.cc:654] T 34c4e61c8a384b5696e55a0dd5b77d4e P 50ddcdd9d41e4c0ea5d0eeae32f6e187: Neither blocks nor log segments found. Creating new log.
I20260806 03:03:26.746019 16149 log.cc:824] T 34c4e61c8a384b5696e55a0dd5b77d4e P 50ddcdd9d41e4c0ea5d0eeae32f6e187: Log is configured to *not* fsync() on all Append() calls
I20260806 03:03:26.748052 16146 raft_consensus.cc:348] T 34c4e61c8a384b5696e55a0dd5b77d4e P 14d8bb5a6aeb49c99ded6a54400fae01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ee24a099afe42d89e3caeb7d9d298a4" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 38719 } } peers { permanent_uuid: "14d8bb5a6aeb49c99ded6a54400fae01" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 35369 } } peers { permanent_uuid: "50ddcdd9d41e4c0ea5d0eeae32f6e187" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 32799 } }
I20260806 03:03:26.748569 16146 raft_consensus.cc:374] T 34c4e61c8a384b5696e55a0dd5b77d4e P 14d8bb5a6aeb49c99ded6a54400fae01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:03:26.748667 16146 raft_consensus.cc:729] T 34c4e61c8a384b5696e55a0dd5b77d4e P 14d8bb5a6aeb49c99ded6a54400fae01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14d8bb5a6aeb49c99ded6a54400fae01, State: Initialized, Role: FOLLOWER
I20260806 03:03:26.748760 16145 raft_consensus.cc:348] T 34c4e61c8a384b5696e55a0dd5b77d4e P 9ee24a099afe42d89e3caeb7d9d298a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ee24a099afe42d89e3caeb7d9d298a4" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 38719 } } peers { permanent_uuid: "14d8bb5a6aeb49c99ded6a54400fae01" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 35369 } } peers { permanent_uuid: "50ddcdd9d41e4c0ea5d0eeae32f6e187" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 32799 } }
I20260806 03:03:26.749221 16145 raft_consensus.cc:374] T 34c4e61c8a384b5696e55a0dd5b77d4e P 9ee24a099afe42d89e3caeb7d9d298a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:03:26.749332 16145 raft_consensus.cc:729] T 34c4e61c8a384b5696e55a0dd5b77d4e P 9ee24a099afe42d89e3caeb7d9d298a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ee24a099afe42d89e3caeb7d9d298a4, State: Initialized, Role: FOLLOWER
I20260806 03:03:26.749390 16146 consensus_queue.cc:260] T 34c4e61c8a384b5696e55a0dd5b77d4e P 14d8bb5a6aeb49c99ded6a54400fae01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ee24a099afe42d89e3caeb7d9d298a4" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 38719 } } peers { permanent_uuid: "14d8bb5a6aeb49c99ded6a54400fae01" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 35369 } } peers { permanent_uuid: "50ddcdd9d41e4c0ea5d0eeae32f6e187" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 32799 } }
I20260806 03:03:26.749928 16145 consensus_queue.cc:260] T 34c4e61c8a384b5696e55a0dd5b77d4e P 9ee24a099afe42d89e3caeb7d9d298a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ee24a099afe42d89e3caeb7d9d298a4" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 38719 } } peers { permanent_uuid: "14d8bb5a6aeb49c99ded6a54400fae01" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 35369 } } peers { permanent_uuid: "50ddcdd9d41e4c0ea5d0eeae32f6e187" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 32799 } }
I20260806 03:03:26.750550 16149 tablet_bootstrap.cc:492] T 34c4e61c8a384b5696e55a0dd5b77d4e P 50ddcdd9d41e4c0ea5d0eeae32f6e187: No bootstrap required, opened a new log
I20260806 03:03:26.750852 15990 heartbeater.cc:503] Master 127.31.74.254:39773 was elected leader, sending a full tablet report...
I20260806 03:03:26.750926 16149 ts_tablet_manager.cc:1397] T 34c4e61c8a384b5696e55a0dd5b77d4e P 50ddcdd9d41e4c0ea5d0eeae32f6e187: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.004s
I20260806 03:03:26.751513 16146 ts_tablet_manager.cc:1428] T 34c4e61c8a384b5696e55a0dd5b77d4e P 14d8bb5a6aeb49c99ded6a54400fae01: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260806 03:03:26.754930 16145 ts_tablet_manager.cc:1428] T 34c4e61c8a384b5696e55a0dd5b77d4e P 9ee24a099afe42d89e3caeb7d9d298a4: Time spent starting tablet: real 0.018s user 0.014s sys 0.006s
I20260806 03:03:26.762487 16149 raft_consensus.cc:348] T 34c4e61c8a384b5696e55a0dd5b77d4e P 50ddcdd9d41e4c0ea5d0eeae32f6e187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ee24a099afe42d89e3caeb7d9d298a4" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 38719 } } peers { permanent_uuid: "14d8bb5a6aeb49c99ded6a54400fae01" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 35369 } } peers { permanent_uuid: "50ddcdd9d41e4c0ea5d0eeae32f6e187" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 32799 } }
I20260806 03:03:26.762991 16149 raft_consensus.cc:374] T 34c4e61c8a384b5696e55a0dd5b77d4e P 50ddcdd9d41e4c0ea5d0eeae32f6e187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:03:26.763134 16149 raft_consensus.cc:729] T 34c4e61c8a384b5696e55a0dd5b77d4e P 50ddcdd9d41e4c0ea5d0eeae32f6e187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50ddcdd9d41e4c0ea5d0eeae32f6e187, State: Initialized, Role: FOLLOWER
I20260806 03:03:26.763803 16149 consensus_queue.cc:260] T 34c4e61c8a384b5696e55a0dd5b77d4e P 50ddcdd9d41e4c0ea5d0eeae32f6e187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ee24a099afe42d89e3caeb7d9d298a4" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 38719 } } peers { permanent_uuid: "14d8bb5a6aeb49c99ded6a54400fae01" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 35369 } } peers { permanent_uuid: "50ddcdd9d41e4c0ea5d0eeae32f6e187" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 32799 } }
I20260806 03:03:26.765461 16126 heartbeater.cc:503] Master 127.31.74.254:39773 was elected leader, sending a full tablet report...
I20260806 03:03:26.766057 16149 ts_tablet_manager.cc:1428] T 34c4e61c8a384b5696e55a0dd5b77d4e P 50ddcdd9d41e4c0ea5d0eeae32f6e187: Time spent starting tablet: real 0.015s user 0.011s sys 0.003s
W20260806 03:03:26.832466 16127 tablet.cc:2369] T 34c4e61c8a384b5696e55a0dd5b77d4e P 50ddcdd9d41e4c0ea5d0eeae32f6e187: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 03:03:26.838493 15991 tablet.cc:2369] T 34c4e61c8a384b5696e55a0dd5b77d4e P 14d8bb5a6aeb49c99ded6a54400fae01: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 03:03:26.843530 16153 raft_consensus.cc:482] T 34c4e61c8a384b5696e55a0dd5b77d4e P 50ddcdd9d41e4c0ea5d0eeae32f6e187 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:03:26.843873 16153 raft_consensus.cc:504] T 34c4e61c8a384b5696e55a0dd5b77d4e P 50ddcdd9d41e4c0ea5d0eeae32f6e187 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ee24a099afe42d89e3caeb7d9d298a4" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 38719 } } peers { permanent_uuid: "14d8bb5a6aeb49c99ded6a54400fae01" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 35369 } } peers { permanent_uuid: "50ddcdd9d41e4c0ea5d0eeae32f6e187" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 32799 } }
I20260806 03:03:26.845948 16153 leader_election.cc:288] T 34c4e61c8a384b5696e55a0dd5b77d4e P 50ddcdd9d41e4c0ea5d0eeae32f6e187 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ee24a099afe42d89e3caeb7d9d298a4 (127.31.74.193:38719), 14d8bb5a6aeb49c99ded6a54400fae01 (127.31.74.194:35369)
I20260806 03:03:26.855280 16129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:26.846764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.194:35369 (local address 127.31.74.195:47693)
0806 03:03:26.847028 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:26.847057 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0806 03:03:26.847336 (+ 279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:03:26.847809 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:03:26.847815 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:03:26.847854 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:03:26.848100 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:03:26.848109 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:03:26.849417 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:03:26.849422 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:03:26.851863 (+ 2441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:03:26.851898 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:03:26.854258 (+ 2360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:03:26.854267 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:03:26.854518 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:03:26.854562 (+ 44us) client_negotiation.cc:769] Sending connection context
0806 03:03:26.854720 (+ 158us) client_negotiation.cc:240] Negotiation successful
0806 03:03:26.854900 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"mutex_wait_us":32}
I20260806 03:03:26.855489 16142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:26.846780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.195:47693 (local address 127.31.74.194:35369)
0806 03:03:26.847018 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0806 03:03:26.847025 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 03:03:26.847155 (+ 130us) server_negotiation.cc:407] Connection header received
0806 03:03:26.847550 (+ 395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:03:26.847556 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:03:26.847646 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:03:26.847759 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:03:26.848236 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:26.849248 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:26.852441 (+ 3193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:26.853842 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:26.854033 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:03:26.854130 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0806 03:03:26.855009 (+ 879us) server_negotiation.cc:299] Negotiation successful
0806 03:03:26.855211 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260806 03:03:26.855618 16140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:26.846766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.195:39863 (local address 127.31.74.193:38719)
0806 03:03:26.847225 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0806 03:03:26.847233 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 03:03:26.847887 (+ 654us) server_negotiation.cc:407] Connection header received
0806 03:03:26.848072 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:03:26.848075 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:03:26.848137 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:03:26.848262 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:03:26.848758 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:26.849704 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:26.852839 (+ 3135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:26.854313 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:26.854495 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:03:26.854612 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0806 03:03:26.855180 (+ 568us) server_negotiation.cc:299] Negotiation successful
0806 03:03:26.855347 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300}
I20260806 03:03:26.856271 16154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:26.846897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:38719 (local address 127.31.74.195:39863)
0806 03:03:26.847751 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:26.847771 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 03:03:26.847885 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:03:26.848306 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:03:26.848310 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:03:26.848329 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:03:26.848601 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:03:26.848608 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:03:26.849876 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:03:26.849882 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:03:26.852630 (+ 2748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:03:26.852643 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:03:26.854777 (+ 2134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:03:26.854783 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:03:26.854959 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:03:26.854990 (+ 31us) client_negotiation.cc:769] Sending connection context
0806 03:03:26.855708 (+ 718us) client_negotiation.cc:240] Negotiation successful
0806 03:03:26.855896 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":732,"mutex_wait_us":23,"thread_start_us":546,"threads_started":1}
I20260806 03:03:26.856621 15942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34c4e61c8a384b5696e55a0dd5b77d4e" candidate_uuid: "50ddcdd9d41e4c0ea5d0eeae32f6e187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14d8bb5a6aeb49c99ded6a54400fae01" is_pre_election: true
I20260806 03:03:26.857311 15942 raft_consensus.cc:2454] T 34c4e61c8a384b5696e55a0dd5b77d4e P 14d8bb5a6aeb49c99ded6a54400fae01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50ddcdd9d41e4c0ea5d0eeae32f6e187 in term 0.
I20260806 03:03:26.857749 15803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34c4e61c8a384b5696e55a0dd5b77d4e" candidate_uuid: "50ddcdd9d41e4c0ea5d0eeae32f6e187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ee24a099afe42d89e3caeb7d9d298a4" is_pre_election: true
I20260806 03:03:26.858309 16014 leader_election.cc:302] T 34c4e61c8a384b5696e55a0dd5b77d4e P 50ddcdd9d41e4c0ea5d0eeae32f6e187 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14d8bb5a6aeb49c99ded6a54400fae01, 50ddcdd9d41e4c0ea5d0eeae32f6e187; no voters:
I20260806 03:03:26.858425 15803 raft_consensus.cc:2454] T 34c4e61c8a384b5696e55a0dd5b77d4e P 9ee24a099afe42d89e3caeb7d9d298a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50ddcdd9d41e4c0ea5d0eeae32f6e187 in term 0.
I20260806 03:03:26.858863 16153 raft_consensus.cc:2781] T 34c4e61c8a384b5696e55a0dd5b77d4e P 50ddcdd9d41e4c0ea5d0eeae32f6e187 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:03:26.859048 16153 raft_consensus.cc:482] T 34c4e61c8a384b5696e55a0dd5b77d4e P 50ddcdd9d41e4c0ea5d0eeae32f6e187 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:03:26.859431 16153 raft_consensus.cc:3037] T 34c4e61c8a384b5696e55a0dd5b77d4e P 50ddcdd9d41e4c0ea5d0eeae32f6e187 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:03:26.862566 16153 raft_consensus.cc:504] T 34c4e61c8a384b5696e55a0dd5b77d4e P 50ddcdd9d41e4c0ea5d0eeae32f6e187 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ee24a099afe42d89e3caeb7d9d298a4" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 38719 } } peers { permanent_uuid: "14d8bb5a6aeb49c99ded6a54400fae01" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 35369 } } peers { permanent_uuid: "50ddcdd9d41e4c0ea5d0eeae32f6e187" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 32799 } }
I20260806 03:03:26.863405 16153 leader_election.cc:288] T 34c4e61c8a384b5696e55a0dd5b77d4e P 50ddcdd9d41e4c0ea5d0eeae32f6e187 [CANDIDATE]: Term 1 election: Requested vote from peers 9ee24a099afe42d89e3caeb7d9d298a4 (127.31.74.193:38719), 14d8bb5a6aeb49c99ded6a54400fae01 (127.31.74.194:35369)
I20260806 03:03:26.864248 15803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34c4e61c8a384b5696e55a0dd5b77d4e" candidate_uuid: "50ddcdd9d41e4c0ea5d0eeae32f6e187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ee24a099afe42d89e3caeb7d9d298a4"
I20260806 03:03:26.864248 15942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34c4e61c8a384b5696e55a0dd5b77d4e" candidate_uuid: "50ddcdd9d41e4c0ea5d0eeae32f6e187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14d8bb5a6aeb49c99ded6a54400fae01"
I20260806 03:03:26.864502 15942 raft_consensus.cc:3037] T 34c4e61c8a384b5696e55a0dd5b77d4e P 14d8bb5a6aeb49c99ded6a54400fae01 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:03:26.864501 15803 raft_consensus.cc:3037] T 34c4e61c8a384b5696e55a0dd5b77d4e P 9ee24a099afe42d89e3caeb7d9d298a4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:03:26.868039 15942 raft_consensus.cc:2454] T 34c4e61c8a384b5696e55a0dd5b77d4e P 14d8bb5a6aeb49c99ded6a54400fae01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50ddcdd9d41e4c0ea5d0eeae32f6e187 in term 1.
I20260806 03:03:26.868039 15803 raft_consensus.cc:2454] T 34c4e61c8a384b5696e55a0dd5b77d4e P 9ee24a099afe42d89e3caeb7d9d298a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50ddcdd9d41e4c0ea5d0eeae32f6e187 in term 1.
I20260806 03:03:26.868675 16012 leader_election.cc:302] T 34c4e61c8a384b5696e55a0dd5b77d4e P 50ddcdd9d41e4c0ea5d0eeae32f6e187 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50ddcdd9d41e4c0ea5d0eeae32f6e187, 9ee24a099afe42d89e3caeb7d9d298a4; no voters:
I20260806 03:03:26.869499 16153 raft_consensus.cc:2781] T 34c4e61c8a384b5696e55a0dd5b77d4e P 50ddcdd9d41e4c0ea5d0eeae32f6e187 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:03:26.870278 16153 raft_consensus.cc:686] T 34c4e61c8a384b5696e55a0dd5b77d4e P 50ddcdd9d41e4c0ea5d0eeae32f6e187 [term 1 LEADER]: Becoming Leader. State: Replica: 50ddcdd9d41e4c0ea5d0eeae32f6e187, State: Running, Role: LEADER
I20260806 03:03:26.870930 16153 consensus_queue.cc:237] T 34c4e61c8a384b5696e55a0dd5b77d4e P 50ddcdd9d41e4c0ea5d0eeae32f6e187 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ee24a099afe42d89e3caeb7d9d298a4" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 38719 } } peers { permanent_uuid: "14d8bb5a6aeb49c99ded6a54400fae01" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 35369 } } peers { permanent_uuid: "50ddcdd9d41e4c0ea5d0eeae32f6e187" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 32799 } }
I20260806 03:03:26.877240 15652 catalog_manager.cc:5495] T 34c4e61c8a384b5696e55a0dd5b77d4e P 50ddcdd9d41e4c0ea5d0eeae32f6e187 reported cstate change: term changed from 0 to 1, leader changed from <none> to 50ddcdd9d41e4c0ea5d0eeae32f6e187 (127.31.74.195). New cstate: current_term: 1 leader_uuid: "50ddcdd9d41e4c0ea5d0eeae32f6e187" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ee24a099afe42d89e3caeb7d9d298a4" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 38719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14d8bb5a6aeb49c99ded6a54400fae01" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 35369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50ddcdd9d41e4c0ea5d0eeae32f6e187" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 32799 } health_report { overall_health: HEALTHY } } }
W20260806 03:03:26.891980 15850 tablet.cc:2369] T 34c4e61c8a384b5696e55a0dd5b77d4e P 9ee24a099afe42d89e3caeb7d9d298a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 03:03:26.910557 16144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:26.903303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49938 (local address 127.31.74.195:32799)
0806 03:03:26.903566 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0806 03:03:26.903573 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 03:03:26.903639 (+ 66us) server_negotiation.cc:407] Connection header received
0806 03:03:26.904036 (+ 397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:03:26.904040 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:03:26.904149 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:03:26.904455 (+ 306us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 03:03:26.904853 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:26.905581 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:26.907375 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:26.907832 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:26.907939 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:03:26.908867 (+ 928us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 03:03:26.909574 (+ 707us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 03:03:26.909708 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0806 03:03:26.910191 (+ 483us) server_negotiation.cc:299] Negotiation successful
0806 03:03:26.910313 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 03:03:26.922905 15942 raft_consensus.cc:1261] T 34c4e61c8a384b5696e55a0dd5b77d4e P 14d8bb5a6aeb49c99ded6a54400fae01 [term 1 FOLLOWER]: Refusing update from remote peer 50ddcdd9d41e4c0ea5d0eeae32f6e187: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:03:26.924036 16155 consensus_queue.cc:1035] T 34c4e61c8a384b5696e55a0dd5b77d4e P 50ddcdd9d41e4c0ea5d0eeae32f6e187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14d8bb5a6aeb49c99ded6a54400fae01" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 35369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:03:26.924685 15803 raft_consensus.cc:1261] T 34c4e61c8a384b5696e55a0dd5b77d4e P 9ee24a099afe42d89e3caeb7d9d298a4 [term 1 FOLLOWER]: Refusing update from remote peer 50ddcdd9d41e4c0ea5d0eeae32f6e187: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:03:26.926132 16155 consensus_queue.cc:1035] T 34c4e61c8a384b5696e55a0dd5b77d4e P 50ddcdd9d41e4c0ea5d0eeae32f6e187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ee24a099afe42d89e3caeb7d9d298a4" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 38719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:03:26.945117 16158 mvcc.cc:205] Tried to move back new op lower bound from 7315396226743599104 to 7315396226548027392. Current Snapshot: MvccSnapshot[applied={T|T < 7315396226743599104}]
I20260806 03:03:26.967794 15652 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56252:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 03:03:26.968197 15652 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56252:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 03:03:26.973875 15652 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 37607d7875bc405292c7a75a319e369a: Sending DeleteTablet for 3 replicas of tablet 34c4e61c8a384b5696e55a0dd5b77d4e
I20260806 03:03:26.975533 15782 tablet_service.cc:1514] Processing DeleteTablet for tablet 34c4e61c8a384b5696e55a0dd5b77d4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:03:26 UTC) from {username='kudu', principal='kudu/127.31.74.254@KRBTEST.COM'} at 127.0.0.1:55010
I20260806 03:03:26.975716 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 15716
I20260806 03:03:26.975899 16060 tablet_service.cc:1514] Processing DeleteTablet for tablet 34c4e61c8a384b5696e55a0dd5b77d4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:03:26 UTC) from {username='kudu', principal='kudu/127.31.74.254@KRBTEST.COM'} at 127.0.0.1:49924
I20260806 03:03:26.976620 15921 tablet_service.cc:1514] Processing DeleteTablet for tablet 34c4e61c8a384b5696e55a0dd5b77d4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:03:26 UTC) from {username='kudu', principal='kudu/127.31.74.254@KRBTEST.COM'} at 127.0.0.1:56862
I20260806 03:03:26.985960 16169 tablet_replica.cc:330] T 34c4e61c8a384b5696e55a0dd5b77d4e P 14d8bb5a6aeb49c99ded6a54400fae01: stopping tablet replica
I20260806 03:03:26.987412 16169 raft_consensus.cc:2229] T 34c4e61c8a384b5696e55a0dd5b77d4e P 14d8bb5a6aeb49c99ded6a54400fae01 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:03:26.988785 16169 raft_consensus.cc:2258] T 34c4e61c8a384b5696e55a0dd5b77d4e P 14d8bb5a6aeb49c99ded6a54400fae01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:03:26.992535 16169 ts_tablet_manager.cc:1905] T 34c4e61c8a384b5696e55a0dd5b77d4e P 14d8bb5a6aeb49c99ded6a54400fae01: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:03:26.997084 16167 tablet_replica.cc:330] T 34c4e61c8a384b5696e55a0dd5b77d4e P 9ee24a099afe42d89e3caeb7d9d298a4: stopping tablet replica
I20260806 03:03:26.998100 16167 raft_consensus.cc:2229] T 34c4e61c8a384b5696e55a0dd5b77d4e P 9ee24a099afe42d89e3caeb7d9d298a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:03:26.998947 16167 raft_consensus.cc:2258] T 34c4e61c8a384b5696e55a0dd5b77d4e P 9ee24a099afe42d89e3caeb7d9d298a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:03:27.003958 16169 ts_tablet_manager.cc:1918] T 34c4e61c8a384b5696e55a0dd5b77d4e P 14d8bb5a6aeb49c99ded6a54400fae01: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 03:03:27.004313 16169 log.cc:1196] T 34c4e61c8a384b5696e55a0dd5b77d4e P 14d8bb5a6aeb49c99ded6a54400fae01: Deleting WAL directory at /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-1/wal/wals/34c4e61c8a384b5696e55a0dd5b77d4e
I20260806 03:03:27.005141 16169 ts_tablet_manager.cc:1939] T 34c4e61c8a384b5696e55a0dd5b77d4e P 14d8bb5a6aeb49c99ded6a54400fae01: Deleting consensus metadata
I20260806 03:03:27.006834 16168 tablet_replica.cc:330] T 34c4e61c8a384b5696e55a0dd5b77d4e P 50ddcdd9d41e4c0ea5d0eeae32f6e187: stopping tablet replica
I20260806 03:03:27.007316 16168 raft_consensus.cc:2229] T 34c4e61c8a384b5696e55a0dd5b77d4e P 50ddcdd9d41e4c0ea5d0eeae32f6e187 [term 1 LEADER]: Raft consensus shutting down.
I20260806 03:03:27.007710 15636 catalog_manager.cc:4841] TS 14d8bb5a6aeb49c99ded6a54400fae01 (127.31.74.194:35369): tablet 34c4e61c8a384b5696e55a0dd5b77d4e (table test-table [id=f79950d8da1e43ef979eab10a3abe8db]) successfully deleted
I20260806 03:03:27.008059 16168 raft_consensus.cc:2258] T 34c4e61c8a384b5696e55a0dd5b77d4e P 50ddcdd9d41e4c0ea5d0eeae32f6e187 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:03:27.008762 16167 ts_tablet_manager.cc:1905] T 34c4e61c8a384b5696e55a0dd5b77d4e P 9ee24a099afe42d89e3caeb7d9d298a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:03:27.011480 16168 ts_tablet_manager.cc:1905] T 34c4e61c8a384b5696e55a0dd5b77d4e P 50ddcdd9d41e4c0ea5d0eeae32f6e187: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 06 03:03:27 dist-test-slave-dhph krb5kdc[15605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.74.193@KRBTEST.COM
I20260806 03:03:27.020449 16167 ts_tablet_manager.cc:1918] T 34c4e61c8a384b5696e55a0dd5b77d4e P 9ee24a099afe42d89e3caeb7d9d298a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 03:03:27.020777 16167 log.cc:1196] T 34c4e61c8a384b5696e55a0dd5b77d4e P 9ee24a099afe42d89e3caeb7d9d298a4: Deleting WAL directory at /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-0/wal/wals/34c4e61c8a384b5696e55a0dd5b77d4e
I20260806 03:03:27.021399 16167 ts_tablet_manager.cc:1939] T 34c4e61c8a384b5696e55a0dd5b77d4e P 9ee24a099afe42d89e3caeb7d9d298a4: Deleting consensus metadata
I20260806 03:03:27.023918 15634 catalog_manager.cc:4841] TS 9ee24a099afe42d89e3caeb7d9d298a4 (127.31.74.193:38719): tablet 34c4e61c8a384b5696e55a0dd5b77d4e (table test-table [id=f79950d8da1e43ef979eab10a3abe8db]) successfully deleted
I20260806 03:03:27.024556 16168 ts_tablet_manager.cc:1918] T 34c4e61c8a384b5696e55a0dd5b77d4e P 50ddcdd9d41e4c0ea5d0eeae32f6e187: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 03:03:27.024760 16168 log.cc:1196] T 34c4e61c8a384b5696e55a0dd5b77d4e P 50ddcdd9d41e4c0ea5d0eeae32f6e187: Deleting WAL directory at /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-2/wal/wals/34c4e61c8a384b5696e55a0dd5b77d4e
I20260806 03:03:27.025656 16168 ts_tablet_manager.cc:1939] T 34c4e61c8a384b5696e55a0dd5b77d4e P 50ddcdd9d41e4c0ea5d0eeae32f6e187: Deleting consensus metadata
I20260806 03:03:27.027714 15636 catalog_manager.cc:4841] TS 50ddcdd9d41e4c0ea5d0eeae32f6e187 (127.31.74.195:32799): tablet 34c4e61c8a384b5696e55a0dd5b77d4e (table test-table [id=f79950d8da1e43ef979eab10a3abe8db]) successfully deleted
I20260806 03:03:27.028591 16140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:26.978269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55014 (local address 127.31.74.193:38719)
0806 03:03:26.978538 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0806 03:03:26.978549 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 03:03:26.991716 (+ 13167us) server_negotiation.cc:407] Connection header received
0806 03:03:26.992789 (+ 1073us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:03:26.992803 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:03:26.993252 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:03:26.993485 (+ 233us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:03:27.001310 (+ 7825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:27.002949 (+ 1639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:27.006767 (+ 3818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:27.007500 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:27.007609 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:03:27.017945 (+ 10336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:03:27.017997 (+ 52us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:03:27.018032 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:03:27.018111 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:03:27.023917 (+ 5806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:03:27.025417 (+ 1500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:03:27.025428 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:03:27.025432 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:03:27.025625 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:03:27.026384 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:03:27.026390 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:03:27.026394 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:03:27.027049 (+ 655us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:03:27.027513 (+ 464us) server_negotiation.cc:1035] Waiting for connection context
0806 03:03:27.027947 (+ 434us) server_negotiation.cc:299] Negotiation successful
0806 03:03:27.028198 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260806 03:03:27.029390 15844 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 03:03:27.134068 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 15716
I20260806 03:03:27.162974 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 15855
Aug 06 03:03:27 dist-test-slave-dhph krb5kdc[15605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.74.194@KRBTEST.COM
I20260806 03:03:27.187927 16142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:27.164012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56878 (local address 127.31.74.194:35369)
0806 03:03:27.164262 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0806 03:03:27.164269 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 03:03:27.165100 (+ 831us) server_negotiation.cc:407] Connection header received
0806 03:03:27.165361 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:03:27.165366 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:03:27.165435 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:03:27.165550 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:03:27.167066 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:27.167803 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:27.169087 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:27.169498 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:27.169639 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:03:27.178817 (+ 9178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:03:27.178836 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:03:27.178853 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:03:27.178899 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:03:27.184045 (+ 5146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:03:27.185033 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:03:27.185043 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:03:27.185048 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:03:27.185237 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:03:27.185668 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:03:27.185673 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:03:27.185676 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:03:27.186215 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:03:27.186571 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0806 03:03:27.186949 (+ 378us) server_negotiation.cc:299] Negotiation successful
0806 03:03:27.187334 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260806 03:03:27.188721 15985 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 03:03:27.291497 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 15855
I20260806 03:03:27.313114 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 15996
Aug 06 03:03:27 dist-test-slave-dhph krb5kdc[15605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.74.195@KRBTEST.COM
I20260806 03:03:27.334992 16144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:27.314830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49942 (local address 127.31.74.195:32799)
0806 03:03:27.315093 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0806 03:03:27.315099 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 03:03:27.315121 (+ 22us) server_negotiation.cc:407] Connection header received
0806 03:03:27.315190 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:03:27.315198 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:03:27.315278 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:03:27.315406 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:03:27.316843 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:27.318052 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:27.319781 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:27.320288 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:27.320448 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:03:27.325949 (+ 5501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:03:27.325979 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:03:27.326010 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:03:27.326069 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:03:27.330046 (+ 3977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:03:27.331925 (+ 1879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:03:27.331933 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:03:27.331935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:03:27.332125 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:03:27.332632 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:03:27.332643 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:03:27.332646 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:03:27.333151 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:03:27.333485 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0806 03:03:27.333925 (+ 440us) server_negotiation.cc:299] Negotiation successful
0806 03:03:27.334177 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260806 03:03:27.335886 16121 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 03:03:27.476668 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 15996
I20260806 03:03:27.501494 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 15618
I20260806 03:03:27.502995 15685 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 03:03:27.622318 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 15618
I20260806 03:03:47.652855 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:39773
--webserver_interface=127.31.74.254
--webserver_port=44417
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab
--principal=kudu/127.31.74.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:40437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.74.254:39773
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:03:48.101679 16175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:03:48.102105 16175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:03:48.102286 16175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:03:48.110056 16175 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260806 03:03:48.110235 16175 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260806 03:03:48.110318 16175 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260806 03:03:48.110424 16175 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:03:48.110539 16175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:03:48.110693 16175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:03:48.110782 16175 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:03:48.110869 16175 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:03:48.122897 16175 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:40437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.31.74.254:39773
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab
--principal=kudu/127.31.74.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:39773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.74.254
--webserver_port=44417
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:03:48.124532 16175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:03:48.126276 16175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:03:48.162266 16181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:03:48.162266 16182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:03:48.162644 16184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:03:48.163110 16175 server_base.cc:1034] running on GCE node
I20260806 03:03:48.164292 16175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:03:48.166132 16175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:03:48.167659 16175 hybrid_clock.cc:648] HybridClock initialized: now 1785985428167479 us; error 76 us; skew 500 ppm
I20260806 03:03:48.168314 16175 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 03:03:48 dist-test-slave-dhph krb5kdc[15605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985428, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:03:48.173751 16175 init.cc:377] Logged in from keytab as kudu/127.31.74.254@KRBTEST.COM (short username kudu)
I20260806 03:03:48.176301 16175 webserver.cc:460] Webserver started at http://127.31.74.254:44417/ using document root <none> and password file <none>
I20260806 03:03:48.177282 16175 fs_manager.cc:359] Metadata directory not provided
I20260806 03:03:48.177425 16175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:03:48.186932 16175 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.001s sys 0.004s
I20260806 03:03:48.192041 16191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:03:48.194176 16175 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.002s
I20260806 03:03:48.194396 16175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/master-0/wal
uuid: "37607d7875bc405292c7a75a319e369a"
format_stamp: "Formatted at 2026-08-06 03:03:24 on dist-test-slave-dhph"
server_key: "9e0b3ff9e21e91e8b44ed055d16e0698"
server_key_iv: "d5562264c01f98fce9b14d48a61bf396"
server_key_version: "encryptionkey@0"
I20260806 03:03:48.195766 16175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:03:48.223327 16175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:03:48.230773 16175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:03:48.231098 16175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:03:48.255995 16175 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.254:39773
I20260806 03:03:48.256168 16244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.254:39773 every 8 connection(s)
I20260806 03:03:48.257843 16175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
I20260806 03:03:48.260197 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 16175
I20260806 03:03:48.261480 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.193:38719
--local_ip_for_outbound_sockets=127.31.74.193
--tserver_master_addrs=127.31.74.254:39773
--webserver_port=43289
--webserver_interface=127.31.74.193
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.193.keytab
--principal=kudu/127.31.74.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:40437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 03:03:48.269814 16245 sys_catalog.cc:263] Verifying existing consensus state
I20260806 03:03:48.276153 16245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37607d7875bc405292c7a75a319e369a: Bootstrap starting.
I20260806 03:03:48.300477 16245 log.cc:824] T 00000000000000000000000000000000 P 37607d7875bc405292c7a75a319e369a: Log is configured to *not* fsync() on all Append() calls
I20260806 03:03:48.314852 16245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37607d7875bc405292c7a75a319e369a: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 03:03:48.315505 16245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37607d7875bc405292c7a75a319e369a: Bootstrap complete.
I20260806 03:03:48.327829 16245 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37607d7875bc405292c7a75a319e369a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37607d7875bc405292c7a75a319e369a" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 39773 } }
I20260806 03:03:48.328840 16245 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37607d7875bc405292c7a75a319e369a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37607d7875bc405292c7a75a319e369a, State: Initialized, Role: FOLLOWER
I20260806 03:03:48.329576 16245 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37607d7875bc405292c7a75a319e369a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37607d7875bc405292c7a75a319e369a" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 39773 } }
I20260806 03:03:48.329864 16245 raft_consensus.cc:388] T 00000000000000000000000000000000 P 37607d7875bc405292c7a75a319e369a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:03:48.330016 16245 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37607d7875bc405292c7a75a319e369a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:03:48.330206 16245 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37607d7875bc405292c7a75a319e369a [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:03:48.336921 16245 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37607d7875bc405292c7a75a319e369a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37607d7875bc405292c7a75a319e369a" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 39773 } }
I20260806 03:03:48.337904 16245 leader_election.cc:302] T 00000000000000000000000000000000 P 37607d7875bc405292c7a75a319e369a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37607d7875bc405292c7a75a319e369a; no voters:
I20260806 03:03:48.338758 16245 leader_election.cc:288] T 00000000000000000000000000000000 P 37607d7875bc405292c7a75a319e369a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 03:03:48.338866 16249 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37607d7875bc405292c7a75a319e369a [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:03:48.341568 16245 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37607d7875bc405292c7a75a319e369a [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:03:48.346242 16249 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37607d7875bc405292c7a75a319e369a [term 2 LEADER]: Becoming Leader. State: Replica: 37607d7875bc405292c7a75a319e369a, State: Running, Role: LEADER
I20260806 03:03:48.347498 16249 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37607d7875bc405292c7a75a319e369a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37607d7875bc405292c7a75a319e369a" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 39773 } }
I20260806 03:03:48.351743 16250 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37607d7875bc405292c7a75a319e369a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "37607d7875bc405292c7a75a319e369a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37607d7875bc405292c7a75a319e369a" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 39773 } } }
I20260806 03:03:48.352257 16250 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37607d7875bc405292c7a75a319e369a [sys.catalog]: This master's current role is: LEADER
I20260806 03:03:48.352357 16251 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37607d7875bc405292c7a75a319e369a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37607d7875bc405292c7a75a319e369a. Latest consensus state: current_term: 2 leader_uuid: "37607d7875bc405292c7a75a319e369a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37607d7875bc405292c7a75a319e369a" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 39773 } } }
I20260806 03:03:48.353646 16251 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37607d7875bc405292c7a75a319e369a [sys.catalog]: This master's current role is: LEADER
I20260806 03:03:48.354902 16259 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:03:48.371604 16259 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:03:48.374460 16259 catalog_manager.cc:1259] Loaded cluster ID: be8c261140b64d3db0b57e646b399eca
I20260806 03:03:48.375461 16259 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:03:48.382488 16259 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:03:48.384722 16259 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 37607d7875bc405292c7a75a319e369a: Loaded TSK: 0
I20260806 03:03:48.419562 16259 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 37607d7875bc405292c7a75a319e369a: Generated new TSK 1
I20260806 03:03:48.420032 16259 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 03:03:48.729218 16247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:03:48.729669 16247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:03:48.729887 16247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:03:48.740204 16247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:03:48.740439 16247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:03:48.740696 16247 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.193
I20260806 03:03:48.753875 16247 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:40437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.193.keytab
--principal=kudu/127.31.74.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.193:38719
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.74.193
--webserver_port=43289
--tserver_master_addrs=127.31.74.254:39773
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.193
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:03:48.757130 16247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:03:48.758848 16247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:03:48.776903 16272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:03:48.776840 16273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:03:48.777726 16275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:03:48.779608 16247 server_base.cc:1034] running on GCE node
I20260806 03:03:48.780848 16247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:03:48.783778 16247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:03:48.787307 16247 hybrid_clock.cc:648] HybridClock initialized: now 1785985428787248 us; error 221 us; skew 500 ppm
I20260806 03:03:48.788460 16247 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 03:03:48 dist-test-slave-dhph krb5kdc[15605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985428, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:03:48.793088 16247 init.cc:377] Logged in from keytab as kudu/127.31.74.193@KRBTEST.COM (short username kudu)
I20260806 03:03:48.795197 16247 webserver.cc:460] Webserver started at http://127.31.74.193:43289/ using document root <none> and password file <none>
I20260806 03:03:48.796483 16247 fs_manager.cc:359] Metadata directory not provided
I20260806 03:03:48.796687 16247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:03:48.805680 16247 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 03:03:48.809067 16282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:03:48.810773 16247 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 03:03:48.810981 16247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-0/wal
uuid: "9ee24a099afe42d89e3caeb7d9d298a4"
format_stamp: "Formatted at 2026-08-06 03:03:25 on dist-test-slave-dhph"
server_key: "0a2a669a625632fc1eeeb020390b2e85"
server_key_iv: "0a40ee273a22047da0adebd449b0326d"
server_key_version: "encryptionkey@0"
I20260806 03:03:48.812050 16247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:03:48.834867 16247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:03:48.842726 16247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:03:48.843009 16247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:03:48.844400 16247 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:03:48.846560 16247 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:03:48.846781 16247 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260806 03:03:48.846938 16247 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:03:48.847043 16247 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:03:48.902792 16247 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.193:38719
I20260806 03:03:48.903991 16396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.193:38719 every 8 connection(s)
I20260806 03:03:48.906687 16247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
I20260806 03:03:48.917850 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 16247
I20260806 03:03:48.920274 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.194:35369
--local_ip_for_outbound_sockets=127.31.74.194
--tserver_master_addrs=127.31.74.254:39773
--webserver_port=40765
--webserver_interface=127.31.74.194
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.194.keytab
--principal=kudu/127.31.74.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:40437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 03:03:48 dist-test-slave-dhph krb5kdc[15605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985428, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.193@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 03:03:48.953572 16399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:48.911367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.193:43755 (local address 127.31.74.254:39773)
0806 03:03:48.915466 (+ 4099us) server_negotiation.cc:206] Beginning negotiation
0806 03:03:48.915496 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0806 03:03:48.917069 (+ 1573us) server_negotiation.cc:407] Connection header received
0806 03:03:48.922129 (+ 5060us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:03:48.922185 (+ 56us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:03:48.923533 (+ 1348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:03:48.924247 (+ 714us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:03:48.928944 (+ 4697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:48.931894 (+ 2950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:48.936149 (+ 4255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:48.937184 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:48.937457 (+ 273us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:03:48.942801 (+ 5344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:03:48.942813 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:03:48.942831 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:03:48.942931 (+ 100us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:03:48.947609 (+ 4678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:03:48.948581 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:03:48.948591 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:03:48.948599 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:03:48.948847 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:03:48.949519 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:03:48.949524 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:03:48.949528 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:03:48.950243 (+ 715us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:03:48.950456 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0806 03:03:48.951472 (+ 1016us) server_negotiation.cc:299] Negotiation successful
0806 03:03:48.951901 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3739,"thread_start_us":3525,"threads_started":1}
I20260806 03:03:48.953665 16400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:48.913741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:39773 (local address 127.31.74.193:43755)
0806 03:03:48.916403 (+ 2662us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:48.916472 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0806 03:03:48.920709 (+ 4237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:03:48.924518 (+ 3809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:03:48.924531 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:03:48.925791 (+ 1260us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:03:48.928474 (+ 2683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:03:48.928520 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:03:48.932320 (+ 3800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:03:48.932335 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:03:48.935849 (+ 3514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:03:48.935883 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:03:48.937520 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:03:48.937529 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:03:48.937827 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:03:48.938725 (+ 898us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:03:48.938899 (+ 174us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:03:48.942352 (+ 3453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:03:48.947842 (+ 5490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:03:48.947855 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:03:48.947876 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:03:48.948404 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:03:48.948992 (+ 588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:03:48.948998 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:03:48.949001 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:03:48.949362 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:03:48.950405 (+ 1043us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:03:48.950414 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:03:48.950836 (+ 422us) client_negotiation.cc:769] Sending connection context
0806 03:03:48.951179 (+ 343us) client_negotiation.cc:240] Negotiation successful
0806 03:03:48.951848 (+ 669us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2144,"thread_start_us":794,"threads_started":1}
I20260806 03:03:48.958058 16397 heartbeater.cc:347] Connected to a master server at 127.31.74.254:39773
I20260806 03:03:48.958477 16397 heartbeater.cc:464] Registering TS with master...
I20260806 03:03:48.959686 16397 heartbeater.cc:511] Master 127.31.74.254:39773 requested a full tablet report, sending...
I20260806 03:03:48.963568 16208 ts_manager.cc:194] Registered new tserver with Master: 9ee24a099afe42d89e3caeb7d9d298a4 (127.31.74.193:38719)
I20260806 03:03:48.965386 16208 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.74.193@KRBTEST.COM'} at 127.31.74.193:43755
W20260806 03:03:49.380816 16401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:03:49.381178 16401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:03:49.381415 16401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:03:49.390009 16401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:03:49.390156 16401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:03:49.390280 16401 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.194
I20260806 03:03:49.402338 16401 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:40437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.194.keytab
--principal=kudu/127.31.74.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.194:35369
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.74.194
--webserver_port=40765
--tserver_master_addrs=127.31.74.254:39773
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.194
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:03:49.403857 16401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:03:49.405211 16401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:03:49.422369 16407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:03:49.422734 16409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:03:49.422696 16406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:03:49.428449 16401 server_base.cc:1034] running on GCE node
I20260806 03:03:49.429785 16401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:03:49.431074 16401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:03:49.432466 16401 hybrid_clock.cc:648] HybridClock initialized: now 1785985429432384 us; error 98 us; skew 500 ppm
I20260806 03:03:49.433036 16401 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 03:03:49 dist-test-slave-dhph krb5kdc[15605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985429, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:03:49.442459 16401 init.cc:377] Logged in from keytab as kudu/127.31.74.194@KRBTEST.COM (short username kudu)
I20260806 03:03:49.444638 16401 webserver.cc:460] Webserver started at http://127.31.74.194:40765/ using document root <none> and password file <none>
I20260806 03:03:49.445482 16401 fs_manager.cc:359] Metadata directory not provided
I20260806 03:03:49.445667 16401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:03:49.452533 16401 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20260806 03:03:49.458078 16417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:03:49.460178 16401 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260806 03:03:49.460445 16401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-1/wal
uuid: "14d8bb5a6aeb49c99ded6a54400fae01"
format_stamp: "Formatted at 2026-08-06 03:03:25 on dist-test-slave-dhph"
server_key: "1b2eaf3a507f0a1fcf401be9467ab5b3"
server_key_iv: "adb89725606df4b004c1ffcd0989a03e"
server_key_version: "encryptionkey@0"
I20260806 03:03:49.461694 16401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:03:49.478137 16401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:03:49.483358 16401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:03:49.483700 16401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:03:49.484982 16401 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:03:49.487161 16401 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:03:49.487308 16401 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:03:49.487463 16401 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:03:49.487741 16401 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:03:49.531941 16401 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.194:35369
I20260806 03:03:49.532007 16530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.194:35369 every 8 connection(s)
I20260806 03:03:49.533851 16401 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
I20260806 03:03:49.540910 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 16401
I20260806 03:03:49.542140 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.195:32799
--local_ip_for_outbound_sockets=127.31.74.195
--tserver_master_addrs=127.31.74.254:39773
--webserver_port=46653
--webserver_interface=127.31.74.195
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.195.keytab
--principal=kudu/127.31.74.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:40437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 03:03:49 dist-test-slave-dhph krb5kdc[15605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985429, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.194@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 03:03:49.563632 16533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:49.537455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.194:39007 (local address 127.31.74.254:39773)
0806 03:03:49.538206 (+ 751us) server_negotiation.cc:206] Beginning negotiation
0806 03:03:49.538213 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 03:03:49.539327 (+ 1114us) server_negotiation.cc:407] Connection header received
0806 03:03:49.542067 (+ 2740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:03:49.542072 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:03:49.542166 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:03:49.542345 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:03:49.545034 (+ 2689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:49.545828 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:49.548882 (+ 3054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:49.549935 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:49.550182 (+ 247us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:03:49.555370 (+ 5188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:03:49.555380 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:03:49.555388 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:03:49.555446 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:03:49.559739 (+ 4293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:03:49.560724 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:03:49.560729 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:03:49.560731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:03:49.560875 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:03:49.561393 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:03:49.561406 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:03:49.561409 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:03:49.561846 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:03:49.562117 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0806 03:03:49.562961 (+ 844us) server_negotiation.cc:299] Negotiation successful
0806 03:03:49.563236 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":434,"threads_started":1}
I20260806 03:03:49.564875 16534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:49.538156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:39773 (local address 127.31.74.194:39007)
0806 03:03:49.539083 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:49.539133 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 03:03:49.541596 (+ 2463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:03:49.542744 (+ 1148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:03:49.542761 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:03:49.543755 (+ 994us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:03:49.544761 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:03:49.544797 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:03:49.546028 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:03:49.546037 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:03:49.548544 (+ 2507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:03:49.548576 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:03:49.550335 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:03:49.550345 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:03:49.550765 (+ 420us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:03:49.551760 (+ 995us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:03:49.551800 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:03:49.555003 (+ 3203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:03:49.559964 (+ 4961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:03:49.559980 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:03:49.559995 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:03:49.560561 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:03:49.560996 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:03:49.561006 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:03:49.561011 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:03:49.561259 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:03:49.562086 (+ 827us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:03:49.562095 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:03:49.562486 (+ 391us) client_negotiation.cc:769] Sending connection context
0806 03:03:49.562847 (+ 361us) client_negotiation.cc:240] Negotiation successful
0806 03:03:49.563439 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":663,"thread_start_us":430,"threads_started":1}
I20260806 03:03:49.566872 16531 heartbeater.cc:347] Connected to a master server at 127.31.74.254:39773
I20260806 03:03:49.567314 16531 heartbeater.cc:464] Registering TS with master...
I20260806 03:03:49.568208 16531 heartbeater.cc:511] Master 127.31.74.254:39773 requested a full tablet report, sending...
I20260806 03:03:49.569871 16208 ts_manager.cc:194] Registered new tserver with Master: 14d8bb5a6aeb49c99ded6a54400fae01 (127.31.74.194:35369)
I20260806 03:03:49.570808 16208 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.74.194@KRBTEST.COM'} at 127.31.74.194:39007
I20260806 03:03:49.971621 16397 heartbeater.cc:503] Master 127.31.74.254:39773 was elected leader, sending a full tablet report...
W20260806 03:03:49.982110 16535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:03:49.982486 16535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:03:49.982686 16535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:03:49.991008 16535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:03:49.991195 16535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:03:49.991375 16535 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.195
I20260806 03:03:50.003377 16535 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:40437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.195.keytab
--principal=kudu/127.31.74.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.195:32799
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.74.195
--webserver_port=46653
--tserver_master_addrs=127.31.74.254:39773
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.195
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:03:50.004932 16535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:03:50.007124 16535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:03:50.029119 16540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:03:50.029119 16543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:03:50.028832 16541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:03:50.030344 16535 server_base.cc:1034] running on GCE node
I20260806 03:03:50.031170 16535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:03:50.032655 16535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:03:50.033986 16535 hybrid_clock.cc:648] HybridClock initialized: now 1785985430033918 us; error 50 us; skew 500 ppm
I20260806 03:03:50.034422 16535 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 03:03:50 dist-test-slave-dhph krb5kdc[15605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985430, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:03:50.038812 16535 init.cc:377] Logged in from keytab as kudu/127.31.74.195@KRBTEST.COM (short username kudu)
I20260806 03:03:50.040750 16535 webserver.cc:460] Webserver started at http://127.31.74.195:46653/ using document root <none> and password file <none>
I20260806 03:03:50.041589 16535 fs_manager.cc:359] Metadata directory not provided
I20260806 03:03:50.041769 16535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:03:50.049257 16535 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 03:03:50.053521 16550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:03:50.055588 16535 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 03:03:50.056190 16535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-2/wal
uuid: "50ddcdd9d41e4c0ea5d0eeae32f6e187"
format_stamp: "Formatted at 2026-08-06 03:03:26 on dist-test-slave-dhph"
server_key: "662ead0260e8c6e2922b512030cc50af"
server_key_iv: "99dccf488dd9e5728c50391ffc23c98c"
server_key_version: "encryptionkey@0"
I20260806 03:03:50.058866 16535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:03:50.087988 16535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:03:50.093653 16535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:03:50.094147 16535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:03:50.095388 16535 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:03:50.096923 16535 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:03:50.097065 16535 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:03:50.097209 16535 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:03:50.097309 16535 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:03:50.150570 16535 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.195:32799
I20260806 03:03:50.150722 16665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.195:32799 every 8 connection(s)
I20260806 03:03:50.152246 16535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
I20260806 03:03:50.159096 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 16535
Aug 06 03:03:50 dist-test-slave-dhph krb5kdc[15605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985430, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.195@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 03:03:50.178836 16671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:50.160331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44046 (local address 127.31.74.254:39773)
0806 03:03:50.161244 (+ 913us) server_negotiation.cc:206] Beginning negotiation
0806 03:03:50.161252 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 03:03:50.161384 (+ 132us) server_negotiation.cc:407] Connection header received
0806 03:03:50.161680 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:03:50.161685 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:03:50.161756 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:03:50.161891 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:03:50.164186 (+ 2295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:50.165132 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:50.167070 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:50.167660 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:50.167819 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:03:50.171265 (+ 3446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:03:50.171273 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:03:50.171277 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:03:50.171333 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:03:50.175045 (+ 3712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:03:50.176107 (+ 1062us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:03:50.176113 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:03:50.176117 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:03:50.176293 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:03:50.176814 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:03:50.176825 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:03:50.176827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:03:50.177174 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:03:50.177385 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0806 03:03:50.178015 (+ 630us) server_negotiation.cc:299] Negotiation successful
0806 03:03:50.178295 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":736,"thread_start_us":381,"threads_started":1}
I20260806 03:03:50.188172 16668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:50.156791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.195:50197 (local address 127.31.74.254:39773)
0806 03:03:50.157424 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0806 03:03:50.157430 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 03:03:50.159223 (+ 1793us) server_negotiation.cc:407] Connection header received
0806 03:03:50.162471 (+ 3248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:03:50.162476 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:03:50.162550 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:03:50.162716 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:03:50.166838 (+ 4122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:50.168118 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:50.171231 (+ 3113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:50.172013 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:50.172187 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:03:50.176924 (+ 4737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:03:50.176929 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:03:50.176933 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:03:50.176968 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:03:50.182261 (+ 5293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:03:50.183974 (+ 1713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:03:50.183979 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:03:50.183981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:03:50.184141 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:03:50.184789 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:03:50.184795 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:03:50.184798 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:03:50.185357 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:03:50.185494 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0806 03:03:50.187609 (+ 2115us) server_negotiation.cc:299] Negotiation successful
0806 03:03:50.187838 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"spinlock_wait_cycles":1792,"thread_start_us":323,"threads_started":1}
I20260806 03:03:50.188819 16669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:50.157898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:39773 (local address 127.31.74.195:50197)
0806 03:03:50.159022 (+ 1124us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:50.159069 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 03:03:50.162130 (+ 3061us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:03:50.163046 (+ 916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:03:50.163057 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:03:50.164404 (+ 1347us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:03:50.166029 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:03:50.166173 (+ 144us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:03:50.168367 (+ 2194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:03:50.168374 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:03:50.170997 (+ 2623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:03:50.171035 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:03:50.172396 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:03:50.172408 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:03:50.172767 (+ 359us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:03:50.173495 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:03:50.173533 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:03:50.176591 (+ 3058us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:03:50.182456 (+ 5865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:03:50.182466 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:03:50.182487 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:03:50.183185 (+ 698us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:03:50.184316 (+ 1131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:03:50.184320 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:03:50.184323 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:03:50.184544 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:03:50.185582 (+ 1038us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:03:50.185593 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:03:50.186451 (+ 858us) client_negotiation.cc:769] Sending connection context
0806 03:03:50.186943 (+ 492us) client_negotiation.cc:240] Negotiation successful
0806 03:03:50.187436 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":795,"thread_start_us":378,"threads_started":1}
I20260806 03:03:50.191851 16666 heartbeater.cc:347] Connected to a master server at 127.31.74.254:39773
I20260806 03:03:50.193506 16666 heartbeater.cc:464] Registering TS with master...
I20260806 03:03:50.194684 16666 heartbeater.cc:511] Master 127.31.74.254:39773 requested a full tablet report, sending...
I20260806 03:03:50.196452 16208 ts_manager.cc:194] Registered new tserver with Master: 50ddcdd9d41e4c0ea5d0eeae32f6e187 (127.31.74.195:32799)
I20260806 03:03:50.197804 16208 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.74.195@KRBTEST.COM'} at 127.31.74.195:50197
I20260806 03:03:50.212878 32043 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 03:03:50.227541 16668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:50.214294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44060 (local address 127.31.74.254:39773)
0806 03:03:50.214507 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0806 03:03:50.214512 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 03:03:50.215057 (+ 545us) server_negotiation.cc:407] Connection header received
0806 03:03:50.215422 (+ 365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:03:50.215427 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:03:50.215599 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:03:50.215747 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 03:03:50.216324 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:50.217796 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:50.221161 (+ 3365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:50.221795 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:50.222267 (+ 472us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:03:50.224798 (+ 2531us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 03:03:50.226254 (+ 1456us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0806 03:03:50.227160 (+ 906us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44060: token expired
Metrics: {"server-negotiator.queue_time_us":76}
W20260806 03:03:50.228363 16668 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44060: token expired
W20260806 03:03:50.228847 16133 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.31.74.254:39773)
I20260806 03:03:50.249231 16668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:50.229777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44064 (local address 127.31.74.254:39773)
0806 03:03:50.229999 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0806 03:03:50.230005 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 03:03:50.230024 (+ 19us) server_negotiation.cc:407] Connection header received
0806 03:03:50.230080 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:03:50.230084 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:03:50.230151 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:03:50.230386 (+ 235us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:03:50.232528 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:50.233412 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:50.236426 (+ 3014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:50.237198 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:50.237355 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:03:50.241190 (+ 3835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:03:50.241196 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:03:50.241200 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:03:50.241283 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:03:50.245429 (+ 4146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:03:50.246462 (+ 1033us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:03:50.246467 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:03:50.246470 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:03:50.246742 (+ 272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:03:50.247205 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:03:50.247210 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:03:50.247216 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:03:50.247811 (+ 595us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:03:50.248056 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0806 03:03:50.248614 (+ 558us) server_negotiation.cc:299] Negotiation successful
0806 03:03:50.248861 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260806 03:03:50.254566 16208 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44064:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 03:03:50.266093 16208 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 03:03:50.308009 16682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:50.293095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47550 (local address 127.31.74.194:35369)
0806 03:03:50.298072 (+ 4977us) server_negotiation.cc:206] Beginning negotiation
0806 03:03:50.298081 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 03:03:50.298334 (+ 253us) server_negotiation.cc:407] Connection header received
0806 03:03:50.298511 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:03:50.298522 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:03:50.298858 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:03:50.299080 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:03:50.299819 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:50.301416 (+ 1597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:50.304248 (+ 2832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:50.305781 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:50.305943 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:03:50.306664 (+ 721us) server_negotiation.cc:1035] Waiting for connection context
0806 03:03:50.307388 (+ 724us) server_negotiation.cc:299] Negotiation successful
0806 03:03:50.307593 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4810,"thread_start_us":4664,"threads_started":1}
I20260806 03:03:50.308159 16679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:50.289698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:38719 (local address 127.0.0.1:36474)
0806 03:03:50.292987 (+ 3289us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:50.293058 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0806 03:03:50.293306 (+ 248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:03:50.296302 (+ 2996us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:03:50.296309 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:03:50.296343 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:03:50.297060 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:03:50.297085 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:03:50.299506 (+ 2421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:03:50.299512 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:03:50.302522 (+ 3010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:03:50.302538 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:03:50.305435 (+ 2897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:03:50.305447 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:03:50.305681 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:03:50.305741 (+ 60us) client_negotiation.cc:769] Sending connection context
0806 03:03:50.305917 (+ 176us) client_negotiation.cc:240] Negotiation successful
0806 03:03:50.306775 (+ 858us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2926,"mutex_wait_us":21,"thread_start_us":7088,"threads_started":1}
I20260806 03:03:50.308678 16677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:50.290051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36474 (local address 127.31.74.193:38719)
0806 03:03:50.295240 (+ 5189us) server_negotiation.cc:206] Beginning negotiation
0806 03:03:50.295250 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 03:03:50.295284 (+ 34us) server_negotiation.cc:407] Connection header received
0806 03:03:50.295393 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:03:50.295403 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:03:50.296021 (+ 618us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:03:50.297008 (+ 987us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:03:50.297506 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:50.299320 (+ 1814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:50.302896 (+ 3576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:50.304724 (+ 1828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:50.304930 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:03:50.305200 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0806 03:03:50.306804 (+ 1604us) server_negotiation.cc:299] Negotiation successful
0806 03:03:50.307893 (+ 1089us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5003,"thread_start_us":4827,"threads_started":1}
I20260806 03:03:50.308929 16678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:50.290091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.195:32799 (local address 127.0.0.1:44176)
0806 03:03:50.297121 (+ 7030us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:50.297147 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 03:03:50.297520 (+ 373us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:03:50.299511 (+ 1991us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:03:50.299515 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:03:50.299547 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:03:50.300847 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:03:50.300862 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:03:50.302698 (+ 1836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:03:50.302704 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:03:50.305386 (+ 2682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:03:50.305402 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:03:50.307922 (+ 2520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:03:50.307929 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:03:50.308200 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:03:50.308240 (+ 40us) client_negotiation.cc:769] Sending connection context
0806 03:03:50.308350 (+ 110us) client_negotiation.cc:240] Negotiation successful
0806 03:03:50.308500 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6881,"mutex_wait_us":5,"thread_start_us":958,"threads_started":1}
I20260806 03:03:50.309055 16680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:50.290741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44176 (local address 127.31.74.195:32799)
0806 03:03:50.298961 (+ 8220us) server_negotiation.cc:206] Beginning negotiation
0806 03:03:50.298970 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 03:03:50.299003 (+ 33us) server_negotiation.cc:407] Connection header received
0806 03:03:50.299092 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:03:50.299099 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:03:50.299296 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:03:50.299644 (+ 348us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:03:50.301318 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:50.302422 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:50.305616 (+ 3194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:50.307363 (+ 1747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:50.307506 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:03:50.307743 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0806 03:03:50.308455 (+ 712us) server_negotiation.cc:299] Negotiation successful
0806 03:03:50.308645 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8056,"thread_start_us":7816,"threads_started":1}
I20260806 03:03:50.310312 16681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:50.292392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.194:35369 (local address 127.0.0.1:47550)
0806 03:03:50.298189 (+ 5797us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:50.298211 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 03:03:50.298339 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:03:50.299065 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:03:50.299070 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:03:50.299098 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:03:50.299562 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:03:50.299574 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:03:50.301575 (+ 2001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:03:50.301580 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:03:50.303800 (+ 2220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:03:50.303815 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:03:50.306767 (+ 2952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:03:50.306775 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:03:50.306957 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:03:50.306996 (+ 39us) client_negotiation.cc:769] Sending connection context
0806 03:03:50.307101 (+ 105us) client_negotiation.cc:240] Negotiation successful
0806 03:03:50.307258 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5637,"mutex_wait_us":25,"thread_start_us":5578,"threads_started":1}
I20260806 03:03:50.313659 16465 tablet_service.cc:1467] Processing CreateTablet for tablet 42b9c3599f594151a28a4b1573b55a9f (DEFAULT_TABLE table=test-table [id=d618403be4f34bfd8500eae5a8c37632]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:03:50.313702 16599 tablet_service.cc:1467] Processing CreateTablet for tablet 42b9c3599f594151a28a4b1573b55a9f (DEFAULT_TABLE table=test-table [id=d618403be4f34bfd8500eae5a8c37632]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:03:50.313696 16331 tablet_service.cc:1467] Processing CreateTablet for tablet 42b9c3599f594151a28a4b1573b55a9f (DEFAULT_TABLE table=test-table [id=d618403be4f34bfd8500eae5a8c37632]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:03:50.315500 16465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42b9c3599f594151a28a4b1573b55a9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:03:50.315937 16331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42b9c3599f594151a28a4b1573b55a9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:03:50.315925 16599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42b9c3599f594151a28a4b1573b55a9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:03:50.333694 16684 tablet_bootstrap.cc:492] T 42b9c3599f594151a28a4b1573b55a9f P 14d8bb5a6aeb49c99ded6a54400fae01: Bootstrap starting.
I20260806 03:03:50.337359 16685 tablet_bootstrap.cc:492] T 42b9c3599f594151a28a4b1573b55a9f P 9ee24a099afe42d89e3caeb7d9d298a4: Bootstrap starting.
I20260806 03:03:50.338490 16684 tablet_bootstrap.cc:654] T 42b9c3599f594151a28a4b1573b55a9f P 14d8bb5a6aeb49c99ded6a54400fae01: Neither blocks nor log segments found. Creating new log.
I20260806 03:03:50.340131 16684 log.cc:824] T 42b9c3599f594151a28a4b1573b55a9f P 14d8bb5a6aeb49c99ded6a54400fae01: Log is configured to *not* fsync() on all Append() calls
I20260806 03:03:50.340106 16686 tablet_bootstrap.cc:492] T 42b9c3599f594151a28a4b1573b55a9f P 50ddcdd9d41e4c0ea5d0eeae32f6e187: Bootstrap starting.
I20260806 03:03:50.342065 16685 tablet_bootstrap.cc:654] T 42b9c3599f594151a28a4b1573b55a9f P 9ee24a099afe42d89e3caeb7d9d298a4: Neither blocks nor log segments found. Creating new log.
I20260806 03:03:50.344686 16685 log.cc:824] T 42b9c3599f594151a28a4b1573b55a9f P 9ee24a099afe42d89e3caeb7d9d298a4: Log is configured to *not* fsync() on all Append() calls
I20260806 03:03:50.344965 16686 tablet_bootstrap.cc:654] T 42b9c3599f594151a28a4b1573b55a9f P 50ddcdd9d41e4c0ea5d0eeae32f6e187: Neither blocks nor log segments found. Creating new log.
I20260806 03:03:50.346071 16684 tablet_bootstrap.cc:492] T 42b9c3599f594151a28a4b1573b55a9f P 14d8bb5a6aeb49c99ded6a54400fae01: No bootstrap required, opened a new log
I20260806 03:03:50.346455 16684 ts_tablet_manager.cc:1397] T 42b9c3599f594151a28a4b1573b55a9f P 14d8bb5a6aeb49c99ded6a54400fae01: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.000s
I20260806 03:03:50.346549 16686 log.cc:824] T 42b9c3599f594151a28a4b1573b55a9f P 50ddcdd9d41e4c0ea5d0eeae32f6e187: Log is configured to *not* fsync() on all Append() calls
I20260806 03:03:50.350270 16686 tablet_bootstrap.cc:492] T 42b9c3599f594151a28a4b1573b55a9f P 50ddcdd9d41e4c0ea5d0eeae32f6e187: No bootstrap required, opened a new log
I20260806 03:03:50.350296 16685 tablet_bootstrap.cc:492] T 42b9c3599f594151a28a4b1573b55a9f P 9ee24a099afe42d89e3caeb7d9d298a4: No bootstrap required, opened a new log
I20260806 03:03:50.351075 16685 ts_tablet_manager.cc:1397] T 42b9c3599f594151a28a4b1573b55a9f P 9ee24a099afe42d89e3caeb7d9d298a4: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.007s
I20260806 03:03:50.351279 16686 ts_tablet_manager.cc:1397] T 42b9c3599f594151a28a4b1573b55a9f P 50ddcdd9d41e4c0ea5d0eeae32f6e187: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.004s
I20260806 03:03:50.360172 16684 raft_consensus.cc:348] T 42b9c3599f594151a28a4b1573b55a9f P 14d8bb5a6aeb49c99ded6a54400fae01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ee24a099afe42d89e3caeb7d9d298a4" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 38719 } } peers { permanent_uuid: "50ddcdd9d41e4c0ea5d0eeae32f6e187" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 32799 } } peers { permanent_uuid: "14d8bb5a6aeb49c99ded6a54400fae01" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 35369 } }
I20260806 03:03:50.360733 16684 raft_consensus.cc:374] T 42b9c3599f594151a28a4b1573b55a9f P 14d8bb5a6aeb49c99ded6a54400fae01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:03:50.360862 16684 raft_consensus.cc:729] T 42b9c3599f594151a28a4b1573b55a9f P 14d8bb5a6aeb49c99ded6a54400fae01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14d8bb5a6aeb49c99ded6a54400fae01, State: Initialized, Role: FOLLOWER
I20260806 03:03:50.361218 16686 raft_consensus.cc:348] T 42b9c3599f594151a28a4b1573b55a9f P 50ddcdd9d41e4c0ea5d0eeae32f6e187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ee24a099afe42d89e3caeb7d9d298a4" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 38719 } } peers { permanent_uuid: "50ddcdd9d41e4c0ea5d0eeae32f6e187" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 32799 } } peers { permanent_uuid: "14d8bb5a6aeb49c99ded6a54400fae01" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 35369 } }
I20260806 03:03:50.361655 16684 consensus_queue.cc:260] T 42b9c3599f594151a28a4b1573b55a9f P 14d8bb5a6aeb49c99ded6a54400fae01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ee24a099afe42d89e3caeb7d9d298a4" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 38719 } } peers { permanent_uuid: "50ddcdd9d41e4c0ea5d0eeae32f6e187" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 32799 } } peers { permanent_uuid: "14d8bb5a6aeb49c99ded6a54400fae01" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 35369 } }
I20260806 03:03:50.362059 16686 raft_consensus.cc:374] T 42b9c3599f594151a28a4b1573b55a9f P 50ddcdd9d41e4c0ea5d0eeae32f6e187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:03:50.362219 16686 raft_consensus.cc:729] T 42b9c3599f594151a28a4b1573b55a9f P 50ddcdd9d41e4c0ea5d0eeae32f6e187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50ddcdd9d41e4c0ea5d0eeae32f6e187, State: Initialized, Role: FOLLOWER
I20260806 03:03:50.363055 16686 consensus_queue.cc:260] T 42b9c3599f594151a28a4b1573b55a9f P 50ddcdd9d41e4c0ea5d0eeae32f6e187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ee24a099afe42d89e3caeb7d9d298a4" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 38719 } } peers { permanent_uuid: "50ddcdd9d41e4c0ea5d0eeae32f6e187" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 32799 } } peers { permanent_uuid: "14d8bb5a6aeb49c99ded6a54400fae01" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 35369 } }
I20260806 03:03:50.363335 16531 heartbeater.cc:503] Master 127.31.74.254:39773 was elected leader, sending a full tablet report...
I20260806 03:03:50.364269 16684 ts_tablet_manager.cc:1428] T 42b9c3599f594151a28a4b1573b55a9f P 14d8bb5a6aeb49c99ded6a54400fae01: Time spent starting tablet: real 0.018s user 0.020s sys 0.000s
I20260806 03:03:50.363948 16685 raft_consensus.cc:348] T 42b9c3599f594151a28a4b1573b55a9f P 9ee24a099afe42d89e3caeb7d9d298a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ee24a099afe42d89e3caeb7d9d298a4" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 38719 } } peers { permanent_uuid: "50ddcdd9d41e4c0ea5d0eeae32f6e187" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 32799 } } peers { permanent_uuid: "14d8bb5a6aeb49c99ded6a54400fae01" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 35369 } }
I20260806 03:03:50.364529 16685 raft_consensus.cc:374] T 42b9c3599f594151a28a4b1573b55a9f P 9ee24a099afe42d89e3caeb7d9d298a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:03:50.364638 16685 raft_consensus.cc:729] T 42b9c3599f594151a28a4b1573b55a9f P 9ee24a099afe42d89e3caeb7d9d298a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ee24a099afe42d89e3caeb7d9d298a4, State: Initialized, Role: FOLLOWER
I20260806 03:03:50.365527 16685 consensus_queue.cc:260] T 42b9c3599f594151a28a4b1573b55a9f P 9ee24a099afe42d89e3caeb7d9d298a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ee24a099afe42d89e3caeb7d9d298a4" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 38719 } } peers { permanent_uuid: "50ddcdd9d41e4c0ea5d0eeae32f6e187" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 32799 } } peers { permanent_uuid: "14d8bb5a6aeb49c99ded6a54400fae01" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 35369 } }
I20260806 03:03:50.372068 16666 heartbeater.cc:503] Master 127.31.74.254:39773 was elected leader, sending a full tablet report...
I20260806 03:03:50.372766 16686 ts_tablet_manager.cc:1428] T 42b9c3599f594151a28a4b1573b55a9f P 50ddcdd9d41e4c0ea5d0eeae32f6e187: Time spent starting tablet: real 0.021s user 0.016s sys 0.003s
I20260806 03:03:50.374101 16685 ts_tablet_manager.cc:1428] T 42b9c3599f594151a28a4b1573b55a9f P 9ee24a099afe42d89e3caeb7d9d298a4: Time spent starting tablet: real 0.023s user 0.020s sys 0.001s
W20260806 03:03:50.405205 16667 tablet.cc:2369] T 42b9c3599f594151a28a4b1573b55a9f P 50ddcdd9d41e4c0ea5d0eeae32f6e187: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 03:03:50.410712 16398 tablet.cc:2369] T 42b9c3599f594151a28a4b1573b55a9f P 9ee24a099afe42d89e3caeb7d9d298a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 03:03:50.537460 16532 tablet.cc:2369] T 42b9c3599f594151a28a4b1573b55a9f P 14d8bb5a6aeb49c99ded6a54400fae01: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 03:03:50.746137 16691 raft_consensus.cc:482] T 42b9c3599f594151a28a4b1573b55a9f P 50ddcdd9d41e4c0ea5d0eeae32f6e187 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:03:50.746496 16691 raft_consensus.cc:504] T 42b9c3599f594151a28a4b1573b55a9f P 50ddcdd9d41e4c0ea5d0eeae32f6e187 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ee24a099afe42d89e3caeb7d9d298a4" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 38719 } } peers { permanent_uuid: "50ddcdd9d41e4c0ea5d0eeae32f6e187" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 32799 } } peers { permanent_uuid: "14d8bb5a6aeb49c99ded6a54400fae01" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 35369 } }
I20260806 03:03:50.748674 16691 leader_election.cc:288] T 42b9c3599f594151a28a4b1573b55a9f P 50ddcdd9d41e4c0ea5d0eeae32f6e187 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ee24a099afe42d89e3caeb7d9d298a4 (127.31.74.193:38719), 14d8bb5a6aeb49c99ded6a54400fae01 (127.31.74.194:35369)
I20260806 03:03:50.760458 16682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:50.749935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.195:59869 (local address 127.31.74.194:35369)
0806 03:03:50.750534 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0806 03:03:50.750541 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 03:03:50.751106 (+ 565us) server_negotiation.cc:407] Connection header received
0806 03:03:50.751238 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:03:50.751244 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:03:50.751326 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:03:50.751445 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:03:50.752556 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:50.754012 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:50.757470 (+ 3458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:50.759033 (+ 1563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:50.759215 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:03:50.759308 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 03:03:50.759987 (+ 679us) server_negotiation.cc:299] Negotiation successful
0806 03:03:50.760155 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419}
I20260806 03:03:50.760530 16677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:50.749246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.195:52083 (local address 127.31.74.193:38719)
0806 03:03:50.749511 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0806 03:03:50.749517 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 03:03:50.750670 (+ 1153us) server_negotiation.cc:407] Connection header received
0806 03:03:50.751016 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:03:50.751021 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:03:50.751131 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:03:50.751268 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:03:50.752436 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:50.753990 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:50.757459 (+ 3469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:50.758997 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:50.759217 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:03:50.759308 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 03:03:50.760071 (+ 763us) server_negotiation.cc:299] Negotiation successful
0806 03:03:50.760223 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260806 03:03:50.760648 16693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:50.748960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:38719 (local address 127.31.74.195:52083)
0806 03:03:50.750465 (+ 1505us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:50.750502 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 03:03:50.750806 (+ 304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:03:50.751307 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:03:50.751314 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:03:50.751352 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:03:50.752048 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:03:50.752095 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:03:50.754197 (+ 2102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:03:50.754204 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:03:50.757174 (+ 2970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:03:50.757197 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:03:50.759391 (+ 2194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:03:50.759530 (+ 139us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:03:50.759822 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:03:50.759875 (+ 53us) client_negotiation.cc:769] Sending connection context
0806 03:03:50.760012 (+ 137us) client_negotiation.cc:240] Negotiation successful
0806 03:03:50.760154 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1306,"mutex_wait_us":215,"spinlock_wait_cycles":28416,"thread_start_us":1437,"threads_started":1}
I20260806 03:03:50.761214 16694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:50.750180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.194:35369 (local address 127.31.74.195:59869)
0806 03:03:50.750975 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0806 03:03:50.750997 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 03:03:50.751126 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:03:50.751475 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:03:50.751482 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:03:50.751532 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:03:50.752273 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:03:50.752319 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:03:50.754197 (+ 1878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:03:50.754204 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:03:50.757175 (+ 2971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:03:50.757197 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:03:50.759589 (+ 2392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:03:50.759595 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:03:50.759785 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:03:50.759824 (+ 39us) client_negotiation.cc:769] Sending connection context
0806 03:03:50.759917 (+ 93us) client_negotiation.cc:240] Negotiation successful
0806 03:03:50.760101 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"mutex_wait_us":254,"thread_start_us":482,"threads_started":1}
I20260806 03:03:50.761868 16351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42b9c3599f594151a28a4b1573b55a9f" candidate_uuid: "50ddcdd9d41e4c0ea5d0eeae32f6e187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ee24a099afe42d89e3caeb7d9d298a4" is_pre_election: true
I20260806 03:03:50.762372 16486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42b9c3599f594151a28a4b1573b55a9f" candidate_uuid: "50ddcdd9d41e4c0ea5d0eeae32f6e187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14d8bb5a6aeb49c99ded6a54400fae01" is_pre_election: true
I20260806 03:03:50.762794 16351 raft_consensus.cc:2454] T 42b9c3599f594151a28a4b1573b55a9f P 9ee24a099afe42d89e3caeb7d9d298a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50ddcdd9d41e4c0ea5d0eeae32f6e187 in term 0.
I20260806 03:03:50.762960 16486 raft_consensus.cc:2454] T 42b9c3599f594151a28a4b1573b55a9f P 14d8bb5a6aeb49c99ded6a54400fae01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50ddcdd9d41e4c0ea5d0eeae32f6e187 in term 0.
I20260806 03:03:50.764001 16553 leader_election.cc:302] T 42b9c3599f594151a28a4b1573b55a9f P 50ddcdd9d41e4c0ea5d0eeae32f6e187 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 14d8bb5a6aeb49c99ded6a54400fae01, 50ddcdd9d41e4c0ea5d0eeae32f6e187, 9ee24a099afe42d89e3caeb7d9d298a4; no voters:
I20260806 03:03:50.764539 16691 raft_consensus.cc:2781] T 42b9c3599f594151a28a4b1573b55a9f P 50ddcdd9d41e4c0ea5d0eeae32f6e187 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:03:50.764731 16691 raft_consensus.cc:482] T 42b9c3599f594151a28a4b1573b55a9f P 50ddcdd9d41e4c0ea5d0eeae32f6e187 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:03:50.764868 16691 raft_consensus.cc:3037] T 42b9c3599f594151a28a4b1573b55a9f P 50ddcdd9d41e4c0ea5d0eeae32f6e187 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:03:50.768429 16691 raft_consensus.cc:504] T 42b9c3599f594151a28a4b1573b55a9f P 50ddcdd9d41e4c0ea5d0eeae32f6e187 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ee24a099afe42d89e3caeb7d9d298a4" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 38719 } } peers { permanent_uuid: "50ddcdd9d41e4c0ea5d0eeae32f6e187" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 32799 } } peers { permanent_uuid: "14d8bb5a6aeb49c99ded6a54400fae01" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 35369 } }
I20260806 03:03:50.769359 16691 leader_election.cc:288] T 42b9c3599f594151a28a4b1573b55a9f P 50ddcdd9d41e4c0ea5d0eeae32f6e187 [CANDIDATE]: Term 1 election: Requested vote from peers 9ee24a099afe42d89e3caeb7d9d298a4 (127.31.74.193:38719), 14d8bb5a6aeb49c99ded6a54400fae01 (127.31.74.194:35369)
I20260806 03:03:50.770064 16351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42b9c3599f594151a28a4b1573b55a9f" candidate_uuid: "50ddcdd9d41e4c0ea5d0eeae32f6e187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ee24a099afe42d89e3caeb7d9d298a4"
I20260806 03:03:50.770195 16486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42b9c3599f594151a28a4b1573b55a9f" candidate_uuid: "50ddcdd9d41e4c0ea5d0eeae32f6e187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14d8bb5a6aeb49c99ded6a54400fae01"
I20260806 03:03:50.770361 16351 raft_consensus.cc:3037] T 42b9c3599f594151a28a4b1573b55a9f P 9ee24a099afe42d89e3caeb7d9d298a4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:03:50.770455 16486 raft_consensus.cc:3037] T 42b9c3599f594151a28a4b1573b55a9f P 14d8bb5a6aeb49c99ded6a54400fae01 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:03:50.774546 16351 raft_consensus.cc:2454] T 42b9c3599f594151a28a4b1573b55a9f P 9ee24a099afe42d89e3caeb7d9d298a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50ddcdd9d41e4c0ea5d0eeae32f6e187 in term 1.
I20260806 03:03:50.774546 16486 raft_consensus.cc:2454] T 42b9c3599f594151a28a4b1573b55a9f P 14d8bb5a6aeb49c99ded6a54400fae01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50ddcdd9d41e4c0ea5d0eeae32f6e187 in term 1.
I20260806 03:03:50.775228 16551 leader_election.cc:302] T 42b9c3599f594151a28a4b1573b55a9f P 50ddcdd9d41e4c0ea5d0eeae32f6e187 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50ddcdd9d41e4c0ea5d0eeae32f6e187, 9ee24a099afe42d89e3caeb7d9d298a4; no voters:
I20260806 03:03:50.775636 16691 raft_consensus.cc:2781] T 42b9c3599f594151a28a4b1573b55a9f P 50ddcdd9d41e4c0ea5d0eeae32f6e187 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:03:50.776196 16691 raft_consensus.cc:686] T 42b9c3599f594151a28a4b1573b55a9f P 50ddcdd9d41e4c0ea5d0eeae32f6e187 [term 1 LEADER]: Becoming Leader. State: Replica: 50ddcdd9d41e4c0ea5d0eeae32f6e187, State: Running, Role: LEADER
I20260806 03:03:50.776803 16691 consensus_queue.cc:237] T 42b9c3599f594151a28a4b1573b55a9f P 50ddcdd9d41e4c0ea5d0eeae32f6e187 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ee24a099afe42d89e3caeb7d9d298a4" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 38719 } } peers { permanent_uuid: "50ddcdd9d41e4c0ea5d0eeae32f6e187" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 32799 } } peers { permanent_uuid: "14d8bb5a6aeb49c99ded6a54400fae01" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 35369 } }
I20260806 03:03:50.782173 16206 catalog_manager.cc:5495] T 42b9c3599f594151a28a4b1573b55a9f P 50ddcdd9d41e4c0ea5d0eeae32f6e187 reported cstate change: term changed from 0 to 1, leader changed from <none> to 50ddcdd9d41e4c0ea5d0eeae32f6e187 (127.31.74.195). New cstate: current_term: 1 leader_uuid: "50ddcdd9d41e4c0ea5d0eeae32f6e187" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ee24a099afe42d89e3caeb7d9d298a4" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 38719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50ddcdd9d41e4c0ea5d0eeae32f6e187" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 32799 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14d8bb5a6aeb49c99ded6a54400fae01" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 35369 } health_report { overall_health: UNKNOWN } } }
I20260806 03:03:50.900620 16701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:50.893232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44182 (local address 127.31.74.195:32799)
0806 03:03:50.893792 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0806 03:03:50.893797 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 03:03:50.893852 (+ 55us) server_negotiation.cc:407] Connection header received
0806 03:03:50.894144 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:03:50.894148 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:03:50.894234 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:03:50.894400 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 03:03:50.895162 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:50.895890 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:50.897711 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:50.898156 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:50.898274 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:03:50.899071 (+ 797us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 03:03:50.899731 (+ 660us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 03:03:50.899912 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0806 03:03:50.900129 (+ 217us) server_negotiation.cc:299] Negotiation successful
0806 03:03:50.900399 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":327,"threads_started":1}
I20260806 03:03:50.918671 16486 raft_consensus.cc:1261] T 42b9c3599f594151a28a4b1573b55a9f P 14d8bb5a6aeb49c99ded6a54400fae01 [term 1 FOLLOWER]: Refusing update from remote peer 50ddcdd9d41e4c0ea5d0eeae32f6e187: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:03:50.919214 16351 raft_consensus.cc:1261] T 42b9c3599f594151a28a4b1573b55a9f P 9ee24a099afe42d89e3caeb7d9d298a4 [term 1 FOLLOWER]: Refusing update from remote peer 50ddcdd9d41e4c0ea5d0eeae32f6e187: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:03:50.919852 16695 consensus_queue.cc:1035] T 42b9c3599f594151a28a4b1573b55a9f P 50ddcdd9d41e4c0ea5d0eeae32f6e187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14d8bb5a6aeb49c99ded6a54400fae01" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 35369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:03:50.920572 16691 consensus_queue.cc:1035] T 42b9c3599f594151a28a4b1573b55a9f P 50ddcdd9d41e4c0ea5d0eeae32f6e187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ee24a099afe42d89e3caeb7d9d298a4" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 38719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:03:50.934487 16702 mvcc.cc:205] Tried to move back new op lower bound from 7315396325027729408 to 7315396324466130944. Current Snapshot: MvccSnapshot[applied={T|T < 7315396325027729408}]
I20260806 03:03:50.936213 16705 mvcc.cc:205] Tried to move back new op lower bound from 7315396325027729408 to 7315396324466130944. Current Snapshot: MvccSnapshot[applied={T|T < 7315396325027729408}]
I20260806 03:03:50.956944 16206 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 03:03:50.957460 16206 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 03:03:50.963346 16206 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 37607d7875bc405292c7a75a319e369a: Sending DeleteTablet for 3 replicas of tablet 42b9c3599f594151a28a4b1573b55a9f
I20260806 03:03:50.964993 16331 tablet_service.cc:1514] Processing DeleteTablet for tablet 42b9c3599f594151a28a4b1573b55a9f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:03:50 UTC) from {username='kudu', principal='kudu/127.31.74.254@KRBTEST.COM'} at 127.0.0.1:36474
I20260806 03:03:50.965721 16465 tablet_service.cc:1514] Processing DeleteTablet for tablet 42b9c3599f594151a28a4b1573b55a9f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:03:50 UTC) from {username='kudu', principal='kudu/127.31.74.254@KRBTEST.COM'} at 127.0.0.1:47550
I20260806 03:03:50.965886 16599 tablet_service.cc:1514] Processing DeleteTablet for tablet 42b9c3599f594151a28a4b1573b55a9f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:03:50 UTC) from {username='kudu', principal='kudu/127.31.74.254@KRBTEST.COM'} at 127.0.0.1:44176
I20260806 03:03:50.967350 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 16247
I20260806 03:03:50.968807 16715 tablet_replica.cc:330] T 42b9c3599f594151a28a4b1573b55a9f P 14d8bb5a6aeb49c99ded6a54400fae01: stopping tablet replica
I20260806 03:03:50.969403 16715 raft_consensus.cc:2229] T 42b9c3599f594151a28a4b1573b55a9f P 14d8bb5a6aeb49c99ded6a54400fae01 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:03:50.969949 16715 raft_consensus.cc:2258] T 42b9c3599f594151a28a4b1573b55a9f P 14d8bb5a6aeb49c99ded6a54400fae01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:03:50.972072 16714 tablet_replica.cc:330] T 42b9c3599f594151a28a4b1573b55a9f P 50ddcdd9d41e4c0ea5d0eeae32f6e187: stopping tablet replica
I20260806 03:03:50.972635 16714 raft_consensus.cc:2229] T 42b9c3599f594151a28a4b1573b55a9f P 50ddcdd9d41e4c0ea5d0eeae32f6e187 [term 1 LEADER]: Raft consensus shutting down.
I20260806 03:03:50.973420 16714 raft_consensus.cc:2258] T 42b9c3599f594151a28a4b1573b55a9f P 50ddcdd9d41e4c0ea5d0eeae32f6e187 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:03:50.973708 16715 ts_tablet_manager.cc:1905] T 42b9c3599f594151a28a4b1573b55a9f P 14d8bb5a6aeb49c99ded6a54400fae01: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:03:50.975381 16714 ts_tablet_manager.cc:1905] T 42b9c3599f594151a28a4b1573b55a9f P 50ddcdd9d41e4c0ea5d0eeae32f6e187: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:03:50.984354 16714 ts_tablet_manager.cc:1918] T 42b9c3599f594151a28a4b1573b55a9f P 50ddcdd9d41e4c0ea5d0eeae32f6e187: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 03:03:50.984650 16714 log.cc:1196] T 42b9c3599f594151a28a4b1573b55a9f P 50ddcdd9d41e4c0ea5d0eeae32f6e187: Deleting WAL directory at /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-2/wal/wals/42b9c3599f594151a28a4b1573b55a9f
I20260806 03:03:50.985561 16714 ts_tablet_manager.cc:1939] T 42b9c3599f594151a28a4b1573b55a9f P 50ddcdd9d41e4c0ea5d0eeae32f6e187: Deleting consensus metadata
I20260806 03:03:50.989789 16194 catalog_manager.cc:4841] TS 50ddcdd9d41e4c0ea5d0eeae32f6e187 (127.31.74.195:32799): tablet 42b9c3599f594151a28a4b1573b55a9f (table test-table [id=d618403be4f34bfd8500eae5a8c37632]) successfully deleted
I20260806 03:03:50.990899 16715 ts_tablet_manager.cc:1918] T 42b9c3599f594151a28a4b1573b55a9f P 14d8bb5a6aeb49c99ded6a54400fae01: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 03:03:50.991173 16715 log.cc:1196] T 42b9c3599f594151a28a4b1573b55a9f P 14d8bb5a6aeb49c99ded6a54400fae01: Deleting WAL directory at /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-1/wal/wals/42b9c3599f594151a28a4b1573b55a9f
I20260806 03:03:50.992601 16715 ts_tablet_manager.cc:1939] T 42b9c3599f594151a28a4b1573b55a9f P 14d8bb5a6aeb49c99ded6a54400fae01: Deleting consensus metadata
I20260806 03:03:50.995649 16713 tablet_replica.cc:330] T 42b9c3599f594151a28a4b1573b55a9f P 9ee24a099afe42d89e3caeb7d9d298a4: stopping tablet replica
I20260806 03:03:50.996621 16713 raft_consensus.cc:2229] T 42b9c3599f594151a28a4b1573b55a9f P 9ee24a099afe42d89e3caeb7d9d298a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:03:50.998396 16194 catalog_manager.cc:4841] TS 14d8bb5a6aeb49c99ded6a54400fae01 (127.31.74.194:35369): tablet 42b9c3599f594151a28a4b1573b55a9f (table test-table [id=d618403be4f34bfd8500eae5a8c37632]) successfully deleted
I20260806 03:03:50.998633 16713 raft_consensus.cc:2258] T 42b9c3599f594151a28a4b1573b55a9f P 9ee24a099afe42d89e3caeb7d9d298a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:03:51.002204 16713 ts_tablet_manager.cc:1905] T 42b9c3599f594151a28a4b1573b55a9f P 9ee24a099afe42d89e3caeb7d9d298a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:03:51.010231 16713 ts_tablet_manager.cc:1918] T 42b9c3599f594151a28a4b1573b55a9f P 9ee24a099afe42d89e3caeb7d9d298a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 03:03:51.010638 16713 log.cc:1196] T 42b9c3599f594151a28a4b1573b55a9f P 9ee24a099afe42d89e3caeb7d9d298a4: Deleting WAL directory at /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-0/wal/wals/42b9c3599f594151a28a4b1573b55a9f
I20260806 03:03:51.012524 16713 ts_tablet_manager.cc:1939] T 42b9c3599f594151a28a4b1573b55a9f P 9ee24a099afe42d89e3caeb7d9d298a4: Deleting consensus metadata
I20260806 03:03:51.015923 16192 catalog_manager.cc:4841] TS 9ee24a099afe42d89e3caeb7d9d298a4 (127.31.74.193:38719): tablet 42b9c3599f594151a28a4b1573b55a9f (table test-table [id=d618403be4f34bfd8500eae5a8c37632]) successfully deleted
I20260806 03:03:51.023833 16677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:50.969119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36482 (local address 127.31.74.193:38719)
0806 03:03:50.969329 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0806 03:03:50.969336 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 03:03:50.983610 (+ 14274us) server_negotiation.cc:407] Connection header received
0806 03:03:50.983990 (+ 380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:03:50.984000 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:03:50.984128 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:03:50.984233 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:03:50.994899 (+ 10666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:50.996599 (+ 1700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:50.999056 (+ 2457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:51.000041 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:51.000621 (+ 580us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:03:51.010562 (+ 9941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:03:51.010614 (+ 52us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:03:51.010639 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:03:51.010717 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:03:51.017804 (+ 7087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:03:51.020271 (+ 2467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:03:51.020284 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:03:51.020286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:03:51.020580 (+ 294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:03:51.021553 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:03:51.021559 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:03:51.021562 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:03:51.022131 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:03:51.022550 (+ 419us) server_negotiation.cc:1035] Waiting for connection context
0806 03:03:51.023037 (+ 487us) server_negotiation.cc:299] Negotiation successful
0806 03:03:51.023317 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260806 03:03:51.025084 16392 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 03:03:51.137331 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 16247
I20260806 03:03:51.178718 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 16401
I20260806 03:03:51.209426 16682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:51.179786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47552 (local address 127.31.74.194:35369)
0806 03:03:51.179964 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0806 03:03:51.179969 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 03:03:51.179988 (+ 19us) server_negotiation.cc:407] Connection header received
0806 03:03:51.180531 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:03:51.180535 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:03:51.180627 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:03:51.180755 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:03:51.187083 (+ 6328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:51.188266 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:51.193185 (+ 4919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:51.193711 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:51.193849 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:03:51.200379 (+ 6530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:03:51.200400 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:03:51.200418 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:03:51.200467 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:03:51.204632 (+ 4165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:03:51.205777 (+ 1145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:03:51.205786 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:03:51.205789 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:03:51.205973 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:03:51.207125 (+ 1152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:03:51.207129 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:03:51.207132 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:03:51.208052 (+ 920us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:03:51.208598 (+ 546us) server_negotiation.cc:1035] Waiting for connection context
0806 03:03:51.208904 (+ 306us) server_negotiation.cc:299] Negotiation successful
0806 03:03:51.209129 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 03:03:51.211498 16526 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 03:03:51.341974 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 16401
I20260806 03:03:51.367384 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 16535
I20260806 03:03:51.388449 16701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:03:51.368646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44192 (local address 127.31.74.195:32799)
0806 03:03:51.368974 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0806 03:03:51.368978 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 03:03:51.368994 (+ 16us) server_negotiation.cc:407] Connection header received
0806 03:03:51.369048 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:03:51.369052 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:03:51.369113 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:03:51.369245 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:03:51.370821 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:51.371623 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:51.373376 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:03:51.373865 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:03:51.374030 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:03:51.379890 (+ 5860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:03:51.379905 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:03:51.379924 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:03:51.379972 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:03:51.384312 (+ 4340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:03:51.385658 (+ 1346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:03:51.385665 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:03:51.385668 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:03:51.385883 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:03:51.386413 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:03:51.386417 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:03:51.386419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:03:51.387008 (+ 589us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:03:51.387240 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0806 03:03:51.387723 (+ 483us) server_negotiation.cc:299] Negotiation successful
0806 03:03:51.388049 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174}
I20260806 03:03:51.389314 16661 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 03:03:51.545132 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 16535
I20260806 03:03:51.575417 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 16175
I20260806 03:03:51.577010 16240 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 03:03:51.718061 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 16175
I20260806 03:04:11.748993 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:39773
--webserver_interface=127.31.74.254
--webserver_port=44417
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab
--principal=kudu/127.31.74.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:40437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.74.254:39773
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:04:12.224985 16721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:04:12.225351 16721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:04:12.225564 16721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:04:12.235493 16721 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260806 03:04:12.235635 16721 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260806 03:04:12.235718 16721 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260806 03:04:12.235807 16721 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:04:12.235906 16721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:04:12.236021 16721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:04:12.236089 16721 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:04:12.236171 16721 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:04:12.248773 16721 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:40437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.31.74.254:39773
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab
--principal=kudu/127.31.74.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:39773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.74.254
--webserver_port=44417
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:04:12.250910 16721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:04:12.253175 16721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:04:12.269243 16726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:04:12.269189 16727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:04:12.269151 16729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:04:12.272760 16721 server_base.cc:1034] running on GCE node
I20260806 03:04:12.273757 16721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:04:12.275691 16721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:04:12.277009 16721 hybrid_clock.cc:648] HybridClock initialized: now 1785985452276951 us; error 45 us; skew 500 ppm
I20260806 03:04:12.277455 16721 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 03:04:12 dist-test-slave-dhph krb5kdc[15605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985452, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:04:12.286202 16721 init.cc:377] Logged in from keytab as kudu/127.31.74.254@KRBTEST.COM (short username kudu)
I20260806 03:04:12.288369 16721 webserver.cc:460] Webserver started at http://127.31.74.254:44417/ using document root <none> and password file <none>
I20260806 03:04:12.289284 16721 fs_manager.cc:359] Metadata directory not provided
I20260806 03:04:12.289459 16721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:04:12.296985 16721 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.000s sys 0.004s
I20260806 03:04:12.300662 16736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:04:12.303015 16721 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260806 03:04:12.303251 16721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/master-0/wal
uuid: "37607d7875bc405292c7a75a319e369a"
format_stamp: "Formatted at 2026-08-06 03:03:24 on dist-test-slave-dhph"
server_key: "9e0b3ff9e21e91e8b44ed055d16e0698"
server_key_iv: "d5562264c01f98fce9b14d48a61bf396"
server_key_version: "encryptionkey@0"
I20260806 03:04:12.304673 16721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:04:12.321492 16721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:04:12.327119 16721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:04:12.327574 16721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:04:12.353176 16721 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.254:39773
I20260806 03:04:12.353178 16788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.254:39773 every 8 connection(s)
I20260806 03:04:12.355580 16721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
I20260806 03:04:12.364198 16789 sys_catalog.cc:263] Verifying existing consensus state
I20260806 03:04:12.366442 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 16721
I20260806 03:04:12.368680 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.193:38719
--local_ip_for_outbound_sockets=127.31.74.193
--tserver_master_addrs=127.31.74.254:39773
--webserver_port=43289
--webserver_interface=127.31.74.193
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.193.keytab
--principal=kudu/127.31.74.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:40437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 03:04:12.369098 16789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37607d7875bc405292c7a75a319e369a: Bootstrap starting.
I20260806 03:04:12.394989 16789 log.cc:824] T 00000000000000000000000000000000 P 37607d7875bc405292c7a75a319e369a: Log is configured to *not* fsync() on all Append() calls
I20260806 03:04:12.421140 16789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37607d7875bc405292c7a75a319e369a: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 03:04:12.421864 16789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37607d7875bc405292c7a75a319e369a: Bootstrap complete.
I20260806 03:04:12.436561 16789 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37607d7875bc405292c7a75a319e369a [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37607d7875bc405292c7a75a319e369a" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 39773 } }
I20260806 03:04:12.437546 16789 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37607d7875bc405292c7a75a319e369a [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37607d7875bc405292c7a75a319e369a, State: Initialized, Role: FOLLOWER
I20260806 03:04:12.438234 16789 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37607d7875bc405292c7a75a319e369a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37607d7875bc405292c7a75a319e369a" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 39773 } }
I20260806 03:04:12.438794 16789 raft_consensus.cc:388] T 00000000000000000000000000000000 P 37607d7875bc405292c7a75a319e369a [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:04:12.438980 16789 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37607d7875bc405292c7a75a319e369a [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:04:12.439217 16789 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37607d7875bc405292c7a75a319e369a [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:04:12.446240 16789 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37607d7875bc405292c7a75a319e369a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37607d7875bc405292c7a75a319e369a" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 39773 } }
I20260806 03:04:12.448851 16789 leader_election.cc:302] T 00000000000000000000000000000000 P 37607d7875bc405292c7a75a319e369a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37607d7875bc405292c7a75a319e369a; no voters:
I20260806 03:04:12.450541 16789 leader_election.cc:288] T 00000000000000000000000000000000 P 37607d7875bc405292c7a75a319e369a [CANDIDATE]: Term 3 election: Requested vote from peers
I20260806 03:04:12.450701 16793 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37607d7875bc405292c7a75a319e369a [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:04:12.453660 16793 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37607d7875bc405292c7a75a319e369a [term 3 LEADER]: Becoming Leader. State: Replica: 37607d7875bc405292c7a75a319e369a, State: Running, Role: LEADER
I20260806 03:04:12.454909 16793 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37607d7875bc405292c7a75a319e369a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37607d7875bc405292c7a75a319e369a" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 39773 } }
I20260806 03:04:12.455224 16789 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37607d7875bc405292c7a75a319e369a [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:04:12.458624 16795 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37607d7875bc405292c7a75a319e369a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37607d7875bc405292c7a75a319e369a. Latest consensus state: current_term: 3 leader_uuid: "37607d7875bc405292c7a75a319e369a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37607d7875bc405292c7a75a319e369a" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 39773 } } }
I20260806 03:04:12.458854 16794 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37607d7875bc405292c7a75a319e369a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "37607d7875bc405292c7a75a319e369a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37607d7875bc405292c7a75a319e369a" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 39773 } } }
I20260806 03:04:12.459324 16795 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37607d7875bc405292c7a75a319e369a [sys.catalog]: This master's current role is: LEADER
I20260806 03:04:12.459388 16794 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37607d7875bc405292c7a75a319e369a [sys.catalog]: This master's current role is: LEADER
I20260806 03:04:12.471681 16801 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:04:12.486207 16801 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:04:12.488430 16801 catalog_manager.cc:1259] Loaded cluster ID: be8c261140b64d3db0b57e646b399eca
I20260806 03:04:12.488665 16801 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:04:12.494768 16801 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:04:12.496867 16801 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 37607d7875bc405292c7a75a319e369a: Loaded TSK: 0
I20260806 03:04:12.497653 16801 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 37607d7875bc405292c7a75a319e369a: Loaded TSK: 1
F20260806 03:04:12.499459 16801 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fcdaa3379ad google::LogMessage::Fail() at ??:0
@ 0x7fcdaa33a88a google::LogMessage::SendToLog() at ??:0
@ 0x7fcdaa3374de google::LogMessage::Flush() at ??:0
@ 0x7fcdaa338a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fcdae09b7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fcdae08a73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fcdbfd661c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fcdbfd65783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fcdbfdf12a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7fcdbfdf10c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7fcdbfe3b5e9 std::function<>::operator()() at ??:0
@ 0x7fcdbfd69cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fcdbfd67c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fcdbfdef610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fcdbfdef439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fcdacf83345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fcdacf8691c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fcdacf86799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fcdacf61eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fcdb023b6db start_thread at ??:0
@ 0x7fcda91d161f clone at ??:0
W20260806 03:04:12.938398 16791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:04:12.939168 16791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:04:12.939337 16791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:04:12.948668 16791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:04:12.948822 16791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:04:12.948956 16791 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.193
I20260806 03:04:12.962456 16791 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:40437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.193.keytab
--principal=kudu/127.31.74.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.193:38719
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.74.193
--webserver_port=43289
--tserver_master_addrs=127.31.74.254:39773
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.193
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:04:12.963999 16791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:04:12.965646 16791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:04:12.982669 16837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:04:12.982717 16836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:04:12.982800 16839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:04:12.984838 16791 server_base.cc:1034] running on GCE node
I20260806 03:04:12.985759 16791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:04:12.987058 16791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:04:12.988448 16791 hybrid_clock.cc:648] HybridClock initialized: now 1785985452988329 us; error 102 us; skew 500 ppm
I20260806 03:04:12.988814 16791 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 03:04:12 dist-test-slave-dhph krb5kdc[15605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985452, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:04:12.992990 16791 init.cc:377] Logged in from keytab as kudu/127.31.74.193@KRBTEST.COM (short username kudu)
I20260806 03:04:12.995114 16791 webserver.cc:460] Webserver started at http://127.31.74.193:43289/ using document root <none> and password file <none>
I20260806 03:04:12.996440 16791 fs_manager.cc:359] Metadata directory not provided
I20260806 03:04:12.996630 16791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:04:13.004156 16791 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20260806 03:04:13.009624 16846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:04:13.012290 16791 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.005s sys 0.000s
I20260806 03:04:13.012660 16791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-0/wal
uuid: "9ee24a099afe42d89e3caeb7d9d298a4"
format_stamp: "Formatted at 2026-08-06 03:03:25 on dist-test-slave-dhph"
server_key: "0a2a669a625632fc1eeeb020390b2e85"
server_key_iv: "0a40ee273a22047da0adebd449b0326d"
server_key_version: "encryptionkey@0"
I20260806 03:04:13.014230 16791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:04:13.038960 16791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:04:13.044283 16791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:04:13.044631 16791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:04:13.045893 16791 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:04:13.047646 16791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:04:13.047768 16791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:04:13.047861 16791 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:04:13.047919 16791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:04:13.093591 16791 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.193:38719
I20260806 03:04:13.093663 16960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.193:38719 every 8 connection(s)
I20260806 03:04:13.095361 16791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
I20260806 03:04:13.100839 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 16791
I20260806 03:04:13.102075 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.194:35369
--local_ip_for_outbound_sockets=127.31.74.194
--tserver_master_addrs=127.31.74.254:39773
--webserver_port=40765
--webserver_interface=127.31.74.194
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.194.keytab
--principal=kudu/127.31.74.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:40437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 03:04:13.102968 16963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:13.099386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:39773 (local address 127.31.74.193:41193)
0806 03:04:13.101223 (+ 1837us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:13.101611 (+ 388us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:39773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1168,"thread_start_us":785,"threads_started":1}
W20260806 03:04:13.104832 16961 heartbeater.cc:644] Failed to heartbeat to 127.31.74.254:39773 (0 consecutive failures): Network error: Failed to ping master at 127.31.74.254:39773: Client connection negotiation failed: client connection to 127.31.74.254:39773: connect: Connection refused (error 111)
I20260806 03:04:13.106201 16963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:13.105661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:39773 (local address 127.31.74.193:38951)
0806 03:04:13.105912 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:13.106022 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:39773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260806 03:04:13.107563 16963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:13.107063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:39773 (local address 127.31.74.193:40153)
0806 03:04:13.107258 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:13.107378 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:39773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260806 03:04:13.107939 16961 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 03:04:13.517020 16964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:04:13.517989 16964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:04:13.518271 16964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:04:13.527715 16964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:04:13.527976 16964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:04:13.528203 16964 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.194
I20260806 03:04:13.543812 16964 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:40437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.194.keytab
--principal=kudu/127.31.74.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.194:35369
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.74.194
--webserver_port=40765
--tserver_master_addrs=127.31.74.254:39773
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.194
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:04:13.545807 16964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:04:13.547307 16964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:04:13.564813 16970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:04:13.565230 16969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:04:13.565202 16972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:04:13.565722 16964 server_base.cc:1034] running on GCE node
I20260806 03:04:13.567162 16964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:04:13.568769 16964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:04:13.570262 16964 hybrid_clock.cc:648] HybridClock initialized: now 1785985453570188 us; error 36 us; skew 500 ppm
I20260806 03:04:13.570989 16964 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 03:04:13 dist-test-slave-dhph krb5kdc[15605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985453, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:04:13.575481 16964 init.cc:377] Logged in from keytab as kudu/127.31.74.194@KRBTEST.COM (short username kudu)
I20260806 03:04:13.577800 16964 webserver.cc:460] Webserver started at http://127.31.74.194:40765/ using document root <none> and password file <none>
I20260806 03:04:13.578969 16964 fs_manager.cc:359] Metadata directory not provided
I20260806 03:04:13.579157 16964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:04:13.586002 16964 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 03:04:13.589536 16980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:04:13.591672 16964 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260806 03:04:13.591928 16964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-1/wal
uuid: "14d8bb5a6aeb49c99ded6a54400fae01"
format_stamp: "Formatted at 2026-08-06 03:03:25 on dist-test-slave-dhph"
server_key: "1b2eaf3a507f0a1fcf401be9467ab5b3"
server_key_iv: "adb89725606df4b004c1ffcd0989a03e"
server_key_version: "encryptionkey@0"
I20260806 03:04:13.593102 16964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:04:13.613242 16964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:04:13.619422 16964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:04:13.619745 16964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:04:13.620831 16964 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:04:13.622872 16964 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:04:13.622978 16964 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:04:13.623116 16964 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:04:13.623176 16964 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:04:13.677591 16964 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.194:35369
I20260806 03:04:13.677657 17097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.194:35369 every 8 connection(s)
I20260806 03:04:13.679327 16964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
I20260806 03:04:13.687042 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 16964
I20260806 03:04:13.687968 17100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:13.684330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:39773 (local address 127.31.74.194:35183)
0806 03:04:13.685574 (+ 1244us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:13.686477 (+ 903us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:39773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":916,"thread_start_us":611,"threads_started":1}
I20260806 03:04:13.687995 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.195:32799
--local_ip_for_outbound_sockets=127.31.74.195
--tserver_master_addrs=127.31.74.254:39773
--webserver_port=46653
--webserver_interface=127.31.74.195
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.195.keytab
--principal=kudu/127.31.74.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:40437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:04:13.689390 17098 heartbeater.cc:644] Failed to heartbeat to 127.31.74.254:39773 (0 consecutive failures): Network error: Failed to ping master at 127.31.74.254:39773: Client connection negotiation failed: client connection to 127.31.74.254:39773: connect: Connection refused (error 111)
I20260806 03:04:13.690724 17100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:13.690128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:39773 (local address 127.31.74.194:52813)
0806 03:04:13.690369 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:13.690497 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:39773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260806 03:04:13.691979 17100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:13.691423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:39773 (local address 127.31.74.194:49989)
0806 03:04:13.691649 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:13.691766 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:39773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260806 03:04:13.692500 17098 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 03:04:14.110249 17102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:14.109163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:39773 (local address 127.31.74.193:47761)
0806 03:04:14.109782 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:14.109999 (+ 217us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:39773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":338,"threads_started":1}
W20260806 03:04:14.133359 17101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:04:14.133705 17101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:04:14.133868 17101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:04:14.144979 17101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:04:14.145144 17101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:04:14.145401 17101 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.195
I20260806 03:04:14.156634 17101 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:40437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.195.keytab
--principal=kudu/127.31.74.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.195:32799
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.74.195
--webserver_port=46653
--tserver_master_addrs=127.31.74.254:39773
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.195
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:04:14.158521 17101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:04:14.160183 17101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:04:14.176268 17107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:04:14.176134 17110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:04:14.176172 17108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:04:14.177347 17101 server_base.cc:1034] running on GCE node
I20260806 03:04:14.178154 17101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:04:14.179641 17101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:04:14.183179 17101 hybrid_clock.cc:648] HybridClock initialized: now 1785985454183011 us; error 108 us; skew 500 ppm
I20260806 03:04:14.183943 17101 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 03:04:14 dist-test-slave-dhph krb5kdc[15605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985454, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:04:14.189457 17101 init.cc:377] Logged in from keytab as kudu/127.31.74.195@KRBTEST.COM (short username kudu)
I20260806 03:04:14.191962 17101 webserver.cc:460] Webserver started at http://127.31.74.195:46653/ using document root <none> and password file <none>
I20260806 03:04:14.193532 17101 fs_manager.cc:359] Metadata directory not provided
I20260806 03:04:14.193670 17101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:04:14.203796 17101 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.003s sys 0.004s
I20260806 03:04:14.210065 17117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:04:14.213455 17101 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.001s
I20260806 03:04:14.213789 17101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-2/wal
uuid: "50ddcdd9d41e4c0ea5d0eeae32f6e187"
format_stamp: "Formatted at 2026-08-06 03:03:26 on dist-test-slave-dhph"
server_key: "662ead0260e8c6e2922b512030cc50af"
server_key_iv: "99dccf488dd9e5728c50391ffc23c98c"
server_key_version: "encryptionkey@0"
I20260806 03:04:14.215406 17101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:04:14.246640 17101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:04:14.252218 17101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:04:14.252511 17101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:04:14.253630 17101 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:04:14.255517 17101 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:04:14.255635 17101 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:04:14.255784 17101 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:04:14.255847 17101 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:04:14.300012 17101 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.195:32799
I20260806 03:04:14.300130 17233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.195:32799 every 8 connection(s)
I20260806 03:04:14.301823 17101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
I20260806 03:04:14.305953 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 17101
I20260806 03:04:14.309211 17236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:14.306867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:39773 (local address 127.31.74.195:58543)
0806 03:04:14.307836 (+ 969us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:14.308145 (+ 309us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:39773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":663,"thread_start_us":419,"threads_started":1}
W20260806 03:04:14.310066 17234 heartbeater.cc:644] Failed to heartbeat to 127.31.74.254:39773 (0 consecutive failures): Network error: Failed to ping master at 127.31.74.254:39773: Client connection negotiation failed: client connection to 127.31.74.254:39773: connect: Connection refused (error 111)
I20260806 03:04:14.311352 17236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:14.310781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:39773 (local address 127.31.74.195:44199)
0806 03:04:14.311101 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:14.311175 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:39773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222}
I20260806 03:04:14.312635 17236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:14.312244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:39773 (local address 127.31.74.195:54453)
0806 03:04:14.312430 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:14.312514 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:39773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260806 03:04:14.312979 17234 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 03:04:14.694289 17238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:14.693314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:39773 (local address 127.31.74.194:54399)
0806 03:04:14.693927 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:14.694041 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:39773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":342,"threads_started":1}
I20260806 03:04:15.113143 17239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:15.111770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:39773 (local address 127.31.74.193:55087)
0806 03:04:15.112585 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:15.112705 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:39773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":643,"thread_start_us":340,"threads_started":1}
I20260806 03:04:15.314851 17240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:15.313792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:39773 (local address 127.31.74.195:36049)
0806 03:04:15.314345 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:15.314457 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:39773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":285,"threads_started":1}
I20260806 03:04:15.696712 17241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:15.695723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:39773 (local address 127.31.74.194:52299)
0806 03:04:15.696366 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:15.696455 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:39773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":310,"threads_started":1}
I20260806 03:04:16.115504 17242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:16.114538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:39773 (local address 127.31.74.193:51093)
0806 03:04:16.115168 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:16.115280 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:39773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":357,"threads_started":1}
I20260806 03:04:16.316861 17243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:16.316053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:39773 (local address 127.31.74.195:35743)
0806 03:04:16.316561 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:16.316646 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:39773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":275,"threads_started":1}
I20260806 03:04:16.699628 17244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:16.698245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:39773 (local address 127.31.74.194:58525)
0806 03:04:16.699259 (+ 1014us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:16.699381 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:39773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":885,"thread_start_us":704,"threads_started":1}
I20260806 03:04:17.118799 17245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:17.117456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:39773 (local address 127.31.74.193:46957)
0806 03:04:17.118266 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:17.118418 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:39773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":344,"threads_started":1}
I20260806 03:04:17.319468 17246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:17.318462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:39773 (local address 127.31.74.195:48533)
0806 03:04:17.319132 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:17.319236 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:39773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":430,"threads_started":1}
I20260806 03:04:17.701843 17247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:17.700904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:39773 (local address 127.31.74.194:36753)
0806 03:04:17.701515 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:17.701629 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:39773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":319,"threads_started":1}
I20260806 03:04:18.121273 17248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:18.120200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:39773 (local address 127.31.74.193:55923)
0806 03:04:18.120746 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:18.121051 (+ 305us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:39773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":280,"threads_started":1}
I20260806 03:04:18.321724 17249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:18.320538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:39773 (local address 127.31.74.195:49505)
0806 03:04:18.321153 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:18.321391 (+ 238us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:39773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":320,"threads_started":1}
I20260806 03:04:18.704275 17250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:18.703404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:39773 (local address 127.31.74.194:34611)
0806 03:04:18.703955 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:18.704046 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:39773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":290,"threads_started":1}
I20260806 03:04:19.123800 17251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:19.122872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:39773 (local address 127.31.74.193:48385)
0806 03:04:19.123524 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:19.123618 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:39773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":370,"threads_started":1}
I20260806 03:04:19.324127 17252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:19.323218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:39773 (local address 127.31.74.195:54183)
0806 03:04:19.323843 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:19.323941 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:39773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":306,"threads_started":1}
I20260806 03:04:19.706691 17253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:19.705639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:39773 (local address 127.31.74.194:56879)
0806 03:04:19.706289 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:19.706419 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:39773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":383,"threads_started":1}
I20260806 03:04:20.126736 17254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:20.125398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:39773 (local address 127.31.74.193:54771)
0806 03:04:20.126081 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:20.126308 (+ 227us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:39773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":347,"threads_started":1}
I20260806 03:04:20.326907 17255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:20.325535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:39773 (local address 127.31.74.195:60641)
0806 03:04:20.326475 (+ 940us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:20.326694 (+ 219us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:39773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":781,"thread_start_us":406,"threads_started":1}
I20260806 03:04:20.709193 17256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:20.708119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:39773 (local address 127.31.74.194:53219)
0806 03:04:20.708865 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:20.708972 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:39773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":378,"threads_started":1}
I20260806 03:04:21.130499 17257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:21.129049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:39773 (local address 127.31.74.193:34181)
0806 03:04:21.129876 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:21.130145 (+ 269us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:39773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":643,"thread_start_us":411,"threads_started":1}
I20260806 03:04:21.330313 17258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:21.329092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:39773 (local address 127.31.74.195:42373)
0806 03:04:21.329964 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:21.330078 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:39773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":728,"thread_start_us":509,"threads_started":1}
I20260806 03:04:21.712309 17259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:21.711017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:39773 (local address 127.31.74.194:46895)
0806 03:04:21.712030 (+ 1013us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:21.712127 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:39773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":912,"thread_start_us":513,"threads_started":1}
I20260806 03:04:22.133904 17260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:22.132089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:39773 (local address 127.31.74.193:42621)
0806 03:04:22.133155 (+ 1066us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:22.133284 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:39773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":773,"thread_start_us":563,"threads_started":1}
I20260806 03:04:22.332639 17261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:22.331774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:39773 (local address 127.31.74.195:38695)
0806 03:04:22.332360 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:22.332455 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:39773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":310,"threads_started":1}
I20260806 03:04:22.714365 17262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:22.713493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:39773 (local address 127.31.74.194:60649)
0806 03:04:22.714092 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:22.714171 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:39773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":329,"threads_started":1}
I20260806 03:04:23.136462 17263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:23.135674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:39773 (local address 127.31.74.193:51481)
0806 03:04:23.136194 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:23.136284 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:39773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":284,"threads_started":1}
I20260806 03:04:23.335029 17264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:23.334093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:39773 (local address 127.31.74.195:41977)
0806 03:04:23.334733 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:23.334839 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:39773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":337,"threads_started":1}
I20260806 03:04:23.716560 17265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:23.715742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:39773 (local address 127.31.74.194:33685)
0806 03:04:23.716284 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:23.716373 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:39773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":279,"threads_started":1}
I20260806 03:04:24.139183 17266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:24.137957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:39773 (local address 127.31.74.193:37237)
0806 03:04:24.138765 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:24.138922 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:39773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":348,"threads_started":1}
I20260806 03:04:24.337833 17267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:24.336710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:39773 (local address 127.31.74.195:53705)
0806 03:04:24.337496 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:24.337600 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:39773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":666,"thread_start_us":571,"threads_started":1}
I20260806 03:04:24.718766 17268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:24.717873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:39773 (local address 127.31.74.194:44705)
0806 03:04:24.718392 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:24.718485 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:39773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":276,"threads_started":1}
I20260806 03:04:25.141721 17269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:25.140661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:39773 (local address 127.31.74.193:44763)
0806 03:04:25.141228 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:25.141322 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:39773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":315,"threads_started":1}
I20260806 03:04:25.340067 17270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:25.339241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:39773 (local address 127.31.74.195:41853)
0806 03:04:25.339763 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:25.339881 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:39773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":254,"threads_started":1}
I20260806 03:04:25.720973 17271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:25.719973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:39773 (local address 127.31.74.194:45055)
0806 03:04:25.720621 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:25.720745 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:39773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":363,"threads_started":1}
I20260806 03:04:26.143908 17272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:26.143010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:39773 (local address 127.31.74.193:48077)
0806 03:04:26.143569 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:26.143690 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:39773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":281,"threads_started":1}
I20260806 03:04:26.342474 17273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:26.341384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:39773 (local address 127.31.74.195:38665)
0806 03:04:26.342116 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:26.342246 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:39773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":359,"threads_started":1}
I20260806 03:04:26.723353 17274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:26.722326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:39773 (local address 127.31.74.194:49695)
0806 03:04:26.722982 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:26.723082 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:39773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":407,"threads_started":1}
I20260806 03:04:27.146178 17275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:27.145193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:39773 (local address 127.31.74.193:39871)
0806 03:04:27.145815 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:27.145936 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:39773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":343,"threads_started":1}
I20260806 03:04:27.344761 17276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:27.343789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:39773 (local address 127.31.74.195:46733)
0806 03:04:27.344407 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:27.344530 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:39773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":320,"threads_started":1}
I20260806 03:04:27.725234 17277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:27.724388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:39773 (local address 127.31.74.194:47291)
0806 03:04:27.724935 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:27.725025 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:39773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":305,"threads_started":1}
I20260806 03:04:28.149667 17278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:28.148426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:39773 (local address 127.31.74.193:45669)
0806 03:04:28.149347 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:28.149457 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:39773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":764,"thread_start_us":633,"threads_started":1}
I20260806 03:04:28.346889 17279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:28.345957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:39773 (local address 127.31.74.195:60725)
0806 03:04:28.346484 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:28.346645 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:39773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":269,"threads_started":1}
I20260806 03:04:28.728104 17280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:28.726933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:39773 (local address 127.31.74.194:37555)
0806 03:04:28.727579 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:28.727862 (+ 283us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:39773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":307,"threads_started":1}
I20260806 03:04:29.152405 17281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:29.151209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:39773 (local address 127.31.74.193:56483)
0806 03:04:29.152108 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:29.152222 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:39773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":776,"thread_start_us":475,"threads_started":1}
I20260806 03:04:29.318290 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 16791
I20260806 03:04:29.342996 17282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:29.319990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60362 (local address 127.31.74.193:38719)
0806 03:04:29.321146 (+ 1156us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:29.321168 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:29.321263 (+ 95us) server_negotiation.cc:407] Connection header received
0806 03:04:29.323411 (+ 2148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:29.323423 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:29.323891 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:29.324291 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:04:29.326149 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:29.327673 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:29.329479 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:29.330266 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:29.330498 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:29.335141 (+ 4643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:04:29.335152 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:04:29.335171 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:04:29.335224 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:04:29.339107 (+ 3883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:04:29.339985 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:04:29.339994 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:04:29.340002 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:04:29.340194 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:04:29.340615 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:04:29.340620 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:04:29.340623 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:04:29.341230 (+ 607us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:04:29.341560 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:29.342044 (+ 484us) server_negotiation.cc:299] Negotiation successful
0806 03:04:29.342534 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":909,"thread_start_us":693,"threads_started":1}
I20260806 03:04:29.344477 16956 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 03:04:29.349895 17284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:29.348686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:39773 (local address 127.31.74.195:48243)
0806 03:04:29.349468 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:29.349655 (+ 187us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.254:39773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":635,"thread_start_us":476,"threads_started":1}
I20260806 03:04:29.444797 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 16791
I20260806 03:04:29.468708 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 16964
I20260806 03:04:29.499279 17285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:29.470298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41782 (local address 127.31.74.194:35369)
0806 03:04:29.471391 (+ 1093us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:29.471418 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:29.471604 (+ 186us) server_negotiation.cc:407] Connection header received
0806 03:04:29.473647 (+ 2043us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:29.473657 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:29.474134 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:29.474470 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:04:29.476485 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:29.477932 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:29.479560 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:29.480396 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:29.480564 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:29.489846 (+ 9282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:04:29.489857 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:04:29.489877 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:04:29.489940 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:04:29.494196 (+ 4256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:04:29.495386 (+ 1190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:04:29.495395 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:04:29.495409 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:04:29.495620 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:04:29.496208 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:04:29.496213 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:04:29.496215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:04:29.496890 (+ 675us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:04:29.497158 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:29.498099 (+ 941us) server_negotiation.cc:299] Negotiation successful
0806 03:04:29.498753 (+ 654us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":831,"thread_start_us":640,"threads_started":1}
I20260806 03:04:29.501123 17093 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 03:04:29.601366 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 16964
I20260806 03:04:29.626425 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 17101
I20260806 03:04:29.650290 17287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:29.627749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34050 (local address 127.31.74.195:32799)
0806 03:04:29.628502 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:29.628518 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:29.628562 (+ 44us) server_negotiation.cc:407] Connection header received
0806 03:04:29.630398 (+ 1836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:29.630408 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:29.631070 (+ 662us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:29.631398 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:04:29.632990 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:29.634461 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:29.636509 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:29.637277 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:29.637443 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:29.641219 (+ 3776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:04:29.641230 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:04:29.641250 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:04:29.641303 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:04:29.645119 (+ 3816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:04:29.646168 (+ 1049us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:04:29.646180 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:04:29.646191 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:04:29.646733 (+ 542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:04:29.647581 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:04:29.647587 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:04:29.647590 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:04:29.648422 (+ 832us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:04:29.648716 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:29.649411 (+ 695us) server_negotiation.cc:299] Negotiation successful
0806 03:04:29.649947 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":383,"threads_started":1}
I20260806 03:04:29.652062 17229 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 03:04:29.748499 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 17101
2026-08-06T03:04:29Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (67914 ms)
[----------] 30 tests from SecurityITest (364430 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 03:04:29 dist-test-slave-dhph krb5kdc[17291](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 03:04:29 dist-test-slave-dhph krb5kdc[17291](info): set up 2 sockets
Aug 06 03:04:29 dist-test-slave-dhph krb5kdc[17291](info): commencing operation
krb5kdc: starting...
W20260806 03:04:31.010799 32043 mini_kdc.cc:121] Time spent starting KDC: real 1.212s user 0.004s sys 0.055s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/krb5kdc/test-user.keytab.
Aug 06 03:04:31 dist-test-slave-dhph krb5kdc[17291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T03:04:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:04:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.74.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab.
Entry for principal kudu/127.31.74.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab.
I20260806 03:04:31.399317 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:45687
--webserver_interface=127.31.74.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab
--principal=kudu/127.31.74.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:45699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.74.254:45687
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:04:31.816247 17305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:04:31.816660 17305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:04:31.816843 17305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:04:31.826316 17305 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:04:31.826545 17305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:04:31.826749 17305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:04:31.826875 17305 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:04:31.826974 17305 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:04:31.838753 17305 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:45699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.74.254:45687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab
--principal=kudu/127.31.74.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:45687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.74.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:04:31.840565 17305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:04:31.842093 17305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:04:31.858768 17310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:04:31.858695 17305 server_base.cc:1034] running on GCE node
W20260806 03:04:31.858656 17311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:04:31.858819 17313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:04:31.860244 17305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:04:31.861986 17305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:04:31.863279 17305 hybrid_clock.cc:648] HybridClock initialized: now 1785985471863221 us; error 40 us; skew 500 ppm
I20260806 03:04:31.863749 17305 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 03:04:31 dist-test-slave-dhph krb5kdc[17291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985471, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:04:31.868341 17305 init.cc:377] Logged in from keytab as kudu/127.31.74.254@KRBTEST.COM (short username kudu)
I20260806 03:04:31.870364 17305 webserver.cc:460] Webserver started at http://127.31.74.254:35441/ using document root <none> and password file <none>
I20260806 03:04:31.871281 17305 fs_manager.cc:359] Metadata directory not provided
I20260806 03:04:31.871440 17305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:04:31.871845 17305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:04:31.875576 17305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/minicluster-data/master-0/data/instance:
uuid: "ab14a05097ef4c97b8636d17618f988a"
format_stamp: "Formatted at 2026-08-06 03:04:31 on dist-test-slave-dhph"
server_key: "e4638baa4fc87f4a33e841e8555d3b2c"
server_key_iv: "72360dbb07f8d87b541a90d25eba47ad"
server_key_version: "encryptionkey@0"
I20260806 03:04:31.876447 17305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/minicluster-data/master-0/wal/instance:
uuid: "ab14a05097ef4c97b8636d17618f988a"
format_stamp: "Formatted at 2026-08-06 03:04:31 on dist-test-slave-dhph"
server_key: "e4638baa4fc87f4a33e841e8555d3b2c"
server_key_iv: "72360dbb07f8d87b541a90d25eba47ad"
server_key_version: "encryptionkey@0"
I20260806 03:04:31.882699 17305 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260806 03:04:31.888093 17320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:04:31.890691 17305 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.007s sys 0.000s
I20260806 03:04:31.891573 17305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/minicluster-data/master-0/wal
uuid: "ab14a05097ef4c97b8636d17618f988a"
format_stamp: "Formatted at 2026-08-06 03:04:31 on dist-test-slave-dhph"
server_key: "e4638baa4fc87f4a33e841e8555d3b2c"
server_key_iv: "72360dbb07f8d87b541a90d25eba47ad"
server_key_version: "encryptionkey@0"
I20260806 03:04:31.892524 17305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:04:31.915261 17305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:04:31.921734 17305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:04:31.922139 17305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:04:31.954325 17305 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.254:45687
I20260806 03:04:31.954344 17374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.254:45687 every 8 connection(s)
I20260806 03:04:31.955955 17305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
I20260806 03:04:31.960631 17375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:04:31.965165 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 17305
I20260806 03:04:31.965420 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/minicluster-data/master-0/wal/instance
I20260806 03:04:31.966150 32043 external_mini_cluster.cc:1428] Setting key ce49a18065e2556019c26bc27f771106
I20260806 03:04:31.975975 17375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab14a05097ef4c97b8636d17618f988a: Bootstrap starting.
I20260806 03:04:31.979966 17375 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab14a05097ef4c97b8636d17618f988a: Neither blocks nor log segments found. Creating new log.
I20260806 03:04:31.981480 17375 log.cc:824] T 00000000000000000000000000000000 P ab14a05097ef4c97b8636d17618f988a: Log is configured to *not* fsync() on all Append() calls
I20260806 03:04:31.984846 17375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab14a05097ef4c97b8636d17618f988a: No bootstrap required, opened a new log
Aug 06 03:04:31 dist-test-slave-dhph krb5kdc[17291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 03:04:31.996896 17375 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab14a05097ef4c97b8636d17618f988a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab14a05097ef4c97b8636d17618f988a" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 45687 } }
I20260806 03:04:31.997500 17375 raft_consensus.cc:374] T 00000000000000000000000000000000 P ab14a05097ef4c97b8636d17618f988a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:04:31.997671 17378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:31.969676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54190 (local address 127.31.74.254:45687)
0806 03:04:31.970778 (+ 1102us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:31.970790 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:31.970845 (+ 55us) server_negotiation.cc:407] Connection header received
0806 03:04:31.973863 (+ 3018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:31.973894 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:31.974369 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:31.975078 (+ 709us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:04:31.977323 (+ 2245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:31.979019 (+ 1696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:31.981043 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:31.981858 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:31.982010 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:31.987041 (+ 5031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:04:31.987053 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:04:31.987075 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:04:31.987137 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:04:31.991816 (+ 4679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:04:31.992976 (+ 1160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:04:31.992990 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:04:31.993000 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:04:31.993234 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:04:31.993756 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:04:31.993761 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:04:31.993765 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:04:31.994645 (+ 880us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:04:31.994992 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:31.995481 (+ 489us) server_negotiation.cc:299] Negotiation successful
0806 03:04:31.996035 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":841,"thread_start_us":603,"threads_started":1}
I20260806 03:04:31.997649 17375 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab14a05097ef4c97b8636d17618f988a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab14a05097ef4c97b8636d17618f988a, State: Initialized, Role: FOLLOWER
I20260806 03:04:31.998344 17375 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab14a05097ef4c97b8636d17618f988a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab14a05097ef4c97b8636d17618f988a" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 45687 } }
I20260806 03:04:31.998641 17375 raft_consensus.cc:388] T 00000000000000000000000000000000 P ab14a05097ef4c97b8636d17618f988a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:04:31.998807 17375 raft_consensus.cc:482] T 00000000000000000000000000000000 P ab14a05097ef4c97b8636d17618f988a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:04:31.999001 17375 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab14a05097ef4c97b8636d17618f988a [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:04:32.002092 17375 raft_consensus.cc:504] T 00000000000000000000000000000000 P ab14a05097ef4c97b8636d17618f988a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab14a05097ef4c97b8636d17618f988a" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 45687 } }
I20260806 03:04:32.002692 17375 leader_election.cc:302] T 00000000000000000000000000000000 P ab14a05097ef4c97b8636d17618f988a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ab14a05097ef4c97b8636d17618f988a; no voters:
I20260806 03:04:32.003388 17375 leader_election.cc:288] T 00000000000000000000000000000000 P ab14a05097ef4c97b8636d17618f988a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:04:32.003551 17380 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ab14a05097ef4c97b8636d17618f988a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:04:32.005273 17380 raft_consensus.cc:686] T 00000000000000000000000000000000 P ab14a05097ef4c97b8636d17618f988a [term 1 LEADER]: Becoming Leader. State: Replica: ab14a05097ef4c97b8636d17618f988a, State: Running, Role: LEADER
I20260806 03:04:32.005934 17380 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab14a05097ef4c97b8636d17618f988a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab14a05097ef4c97b8636d17618f988a" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 45687 } }
I20260806 03:04:32.006201 17375 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab14a05097ef4c97b8636d17618f988a [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:04:32.010551 17381 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab14a05097ef4c97b8636d17618f988a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ab14a05097ef4c97b8636d17618f988a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab14a05097ef4c97b8636d17618f988a" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 45687 } } }
I20260806 03:04:32.011379 17381 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab14a05097ef4c97b8636d17618f988a [sys.catalog]: This master's current role is: LEADER
I20260806 03:04:32.015261 17382 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab14a05097ef4c97b8636d17618f988a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab14a05097ef4c97b8636d17618f988a. Latest consensus state: current_term: 1 leader_uuid: "ab14a05097ef4c97b8636d17618f988a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab14a05097ef4c97b8636d17618f988a" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 45687 } } }
I20260806 03:04:32.016263 17382 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab14a05097ef4c97b8636d17618f988a [sys.catalog]: This master's current role is: LEADER
I20260806 03:04:32.021423 17386 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:04:32.029150 17386 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:04:32.040664 17386 catalog_manager.cc:1347] Generated new cluster ID: 7a403ac7720e4a328a31a9b8ac9deb56
I20260806 03:04:32.040840 17386 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:04:32.060967 17386 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:04:32.062884 17386 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:04:32.073280 17386 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ab14a05097ef4c97b8636d17618f988a: Generated new TSK 0
I20260806 03:04:32.074301 17386 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.74.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.193.keytab.
Entry for principal kudu/127.31.74.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.193.keytab.
I20260806 03:04:32.175702 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.193:0
--local_ip_for_outbound_sockets=127.31.74.193
--webserver_interface=127.31.74.193
--webserver_port=0
--tserver_master_addrs=127.31.74.254:45687
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.193.keytab
--principal=kudu/127.31.74.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:45699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:04:32.578953 17401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:04:32.579329 17401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:04:32.579515 17401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:04:32.588269 17401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:04:32.588471 17401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:04:32.588671 17401 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.193
I20260806 03:04:32.601050 17401 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:45699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.193.keytab
--principal=kudu/127.31.74.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.74.193
--webserver_port=0
--tserver_master_addrs=127.31.74.254:45687
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.193
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:04:32.603457 17401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:04:32.605293 17401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:04:32.621690 17409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:04:32.622141 17406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:04:32.622277 17407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:04:32.623772 17401 server_base.cc:1034] running on GCE node
I20260806 03:04:32.624713 17401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:04:32.625818 17401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:04:32.627179 17401 hybrid_clock.cc:648] HybridClock initialized: now 1785985472627063 us; error 68 us; skew 500 ppm
I20260806 03:04:32.627851 17401 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 03:04:32 dist-test-slave-dhph krb5kdc[17291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985472, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:04:32.631989 17401 init.cc:377] Logged in from keytab as kudu/127.31.74.193@KRBTEST.COM (short username kudu)
I20260806 03:04:32.633879 17401 webserver.cc:460] Webserver started at http://127.31.74.193:45335/ using document root <none> and password file <none>
I20260806 03:04:32.634760 17401 fs_manager.cc:359] Metadata directory not provided
I20260806 03:04:32.634884 17401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:04:32.635262 17401 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:04:32.638464 17401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/minicluster-data/ts-0/data/instance:
uuid: "53bd6348a73246cbb09421ff1cd113d4"
format_stamp: "Formatted at 2026-08-06 03:04:32 on dist-test-slave-dhph"
server_key: "4aa16d9712cf4d5a5ed4132a0b6e996c"
server_key_iv: "395a0ef0c0d35a8f151c3a63f5f9b477"
server_key_version: "encryptionkey@0"
I20260806 03:04:32.639423 17401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/minicluster-data/ts-0/wal/instance:
uuid: "53bd6348a73246cbb09421ff1cd113d4"
format_stamp: "Formatted at 2026-08-06 03:04:32 on dist-test-slave-dhph"
server_key: "4aa16d9712cf4d5a5ed4132a0b6e996c"
server_key_iv: "395a0ef0c0d35a8f151c3a63f5f9b477"
server_key_version: "encryptionkey@0"
I20260806 03:04:32.645533 17401 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260806 03:04:32.650701 17416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:04:32.652693 17401 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260806 03:04:32.652921 17401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/minicluster-data/ts-0/wal
uuid: "53bd6348a73246cbb09421ff1cd113d4"
format_stamp: "Formatted at 2026-08-06 03:04:32 on dist-test-slave-dhph"
server_key: "4aa16d9712cf4d5a5ed4132a0b6e996c"
server_key_iv: "395a0ef0c0d35a8f151c3a63f5f9b477"
server_key_version: "encryptionkey@0"
I20260806 03:04:32.653165 17401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:04:32.679433 17401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:04:32.684350 17401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:04:32.684616 17401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:04:32.685626 17401 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:04:32.687153 17401 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:04:32.687271 17401 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:04:32.687422 17401 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:04:32.687497 17401 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:04:32.728956 17401 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.193:35259
I20260806 03:04:32.729053 17531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.193:35259 every 8 connection(s)
I20260806 03:04:32.730428 17401 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
I20260806 03:04:32.739570 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 17401
I20260806 03:04:32.739889 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/minicluster-data/ts-0/wal/instance
I20260806 03:04:32.741264 32043 external_mini_cluster.cc:1428] Setting key 608b47bd38e5677074fe39002144b346
Aug 06 03:04:32 dist-test-slave-dhph krb5kdc[17291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985472, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.193@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 03:04:32.757779 17534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:32.733614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.193:52993 (local address 127.31.74.254:45687)
0806 03:04:32.734132 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:32.734137 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:32.735428 (+ 1291us) server_negotiation.cc:407] Connection header received
0806 03:04:32.737526 (+ 2098us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:32.737531 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:32.737619 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:32.737952 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:04:32.740439 (+ 2487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:32.741730 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:32.744619 (+ 2889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:32.745247 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:32.745427 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:32.749925 (+ 4498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:04:32.749931 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:04:32.749935 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:04:32.749974 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:04:32.753278 (+ 3304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:04:32.754547 (+ 1269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:04:32.754553 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:04:32.754556 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:04:32.754839 (+ 283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:04:32.755439 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:04:32.755444 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:04:32.755447 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:04:32.755853 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:04:32.755995 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:32.757073 (+ 1078us) server_negotiation.cc:299] Negotiation successful
0806 03:04:32.757387 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":280,"threads_started":1}
I20260806 03:04:32.759215 17535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:32.734222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:45687 (local address 127.31.74.193:52993)
0806 03:04:32.735203 (+ 981us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:32.735277 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0806 03:04:32.737171 (+ 1894us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:04:32.738323 (+ 1152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:04:32.738336 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:04:32.739404 (+ 1068us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:04:32.740139 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:32.740161 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:32.741962 (+ 1801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:32.741966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:32.744292 (+ 2326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:32.744406 (+ 114us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:32.745562 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:32.745568 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:32.745812 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:32.746772 (+ 960us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:04:32.746800 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:04:32.749569 (+ 2769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:04:32.753507 (+ 3938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:04:32.753521 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:04:32.753545 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:04:32.754327 (+ 782us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:04:32.754973 (+ 646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:04:32.754980 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:04:32.754986 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:04:32.755265 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:04:32.756009 (+ 744us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:04:32.756020 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:04:32.756528 (+ 508us) client_negotiation.cc:769] Sending connection context
0806 03:04:32.756869 (+ 341us) client_negotiation.cc:240] Negotiation successful
0806 03:04:32.757536 (+ 667us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":734,"thread_start_us":493,"threads_started":1}
I20260806 03:04:32.761664 17532 heartbeater.cc:347] Connected to a master server at 127.31.74.254:45687
I20260806 03:04:32.762034 17532 heartbeater.cc:464] Registering TS with master...
I20260806 03:04:32.762898 17532 heartbeater.cc:511] Master 127.31.74.254:45687 requested a full tablet report, sending...
I20260806 03:04:32.765215 17337 ts_manager.cc:194] Registered new tserver with Master: 53bd6348a73246cbb09421ff1cd113d4 (127.31.74.193:35259)
I20260806 03:04:32.766885 17337 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.74.193@KRBTEST.COM'} at 127.31.74.193:52993
WARNING: no policy specified for kudu/127.31.74.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.194.keytab.
Entry for principal kudu/127.31.74.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.194.keytab.
I20260806 03:04:32.823994 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.194:0
--local_ip_for_outbound_sockets=127.31.74.194
--webserver_interface=127.31.74.194
--webserver_port=0
--tserver_master_addrs=127.31.74.254:45687
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.194.keytab
--principal=kudu/127.31.74.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:45699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:04:33.231374 17538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:04:33.231700 17538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:04:33.231860 17538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:04:33.239887 17538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:04:33.240026 17538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:04:33.240145 17538 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.194
I20260806 03:04:33.254901 17538 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:45699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.194.keytab
--principal=kudu/127.31.74.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.74.194
--webserver_port=0
--tserver_master_addrs=127.31.74.254:45687
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.194
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:04:33.257833 17538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:04:33.260680 17538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:04:33.280041 17543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:04:33.280251 17546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:04:33.280232 17544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:04:33.281885 17538 server_base.cc:1034] running on GCE node
I20260806 03:04:33.282842 17538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:04:33.284035 17538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:04:33.285359 17538 hybrid_clock.cc:648] HybridClock initialized: now 1785985473285306 us; error 42 us; skew 500 ppm
I20260806 03:04:33.285825 17538 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 03:04:33 dist-test-slave-dhph krb5kdc[17291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985473, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:04:33.289709 17538 init.cc:377] Logged in from keytab as kudu/127.31.74.194@KRBTEST.COM (short username kudu)
I20260806 03:04:33.292179 17538 webserver.cc:460] Webserver started at http://127.31.74.194:45061/ using document root <none> and password file <none>
I20260806 03:04:33.293238 17538 fs_manager.cc:359] Metadata directory not provided
I20260806 03:04:33.293464 17538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:04:33.293983 17538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:04:33.298418 17538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/minicluster-data/ts-1/data/instance:
uuid: "642d05cc9727477e8622f27bdff90e07"
format_stamp: "Formatted at 2026-08-06 03:04:33 on dist-test-slave-dhph"
server_key: "67de938dd0bc505512053c7d2a563aa0"
server_key_iv: "4d8cdd5bcaa651b7b1c56a41af43f971"
server_key_version: "encryptionkey@0"
I20260806 03:04:33.299880 17538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/minicluster-data/ts-1/wal/instance:
uuid: "642d05cc9727477e8622f27bdff90e07"
format_stamp: "Formatted at 2026-08-06 03:04:33 on dist-test-slave-dhph"
server_key: "67de938dd0bc505512053c7d2a563aa0"
server_key_iv: "4d8cdd5bcaa651b7b1c56a41af43f971"
server_key_version: "encryptionkey@0"
I20260806 03:04:33.311169 17538 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.011s sys 0.001s
I20260806 03:04:33.317656 17553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:04:33.320612 17538 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.000s
I20260806 03:04:33.320981 17538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/minicluster-data/ts-1/wal
uuid: "642d05cc9727477e8622f27bdff90e07"
format_stamp: "Formatted at 2026-08-06 03:04:33 on dist-test-slave-dhph"
server_key: "67de938dd0bc505512053c7d2a563aa0"
server_key_iv: "4d8cdd5bcaa651b7b1c56a41af43f971"
server_key_version: "encryptionkey@0"
I20260806 03:04:33.321412 17538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:04:33.345489 17538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:04:33.352979 17538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:04:33.353523 17538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:04:33.355458 17538 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:04:33.358062 17538 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:04:33.358155 17538 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:04:33.358331 17538 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:04:33.358431 17538 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:04:33.408165 17538 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.194:33433
I20260806 03:04:33.408200 17669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.194:33433 every 8 connection(s)
I20260806 03:04:33.411234 17538 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
I20260806 03:04:33.416297 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 17538
I20260806 03:04:33.416524 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/minicluster-data/ts-1/wal/instance
I20260806 03:04:33.417132 32043 external_mini_cluster.cc:1428] Setting key 4df4b9a7fa967a7f382f1657007c108a
Aug 06 03:04:33 dist-test-slave-dhph krb5kdc[17291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985473, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.194@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 03:04:33.442571 17672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:33.414852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.194:51791 (local address 127.31.74.254:45687)
0806 03:04:33.415369 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:33.415374 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:33.416979 (+ 1605us) server_negotiation.cc:407] Connection header received
0806 03:04:33.420107 (+ 3128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:33.420111 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:33.420187 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:33.420320 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:04:33.422471 (+ 2151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:33.423367 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:33.425083 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:33.425568 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:33.425727 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:33.432773 (+ 7046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:04:33.432780 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:04:33.432784 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:04:33.432826 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:04:33.437139 (+ 4313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:04:33.438515 (+ 1376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:04:33.438521 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:04:33.438524 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:04:33.438811 (+ 287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:04:33.439900 (+ 1089us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:04:33.439904 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:04:33.439907 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:04:33.440236 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:04:33.440357 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:33.441779 (+ 1422us) server_negotiation.cc:299] Negotiation successful
0806 03:04:33.442088 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":284,"threads_started":1}
I20260806 03:04:33.444468 17673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:33.415373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:45687 (local address 127.31.74.194:51791)
0806 03:04:33.416794 (+ 1421us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:33.416838 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 03:04:33.419430 (+ 2592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:04:33.420863 (+ 1433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:04:33.420873 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:04:33.421506 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:04:33.422236 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:33.422258 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:33.423565 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:33.423569 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:33.424908 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:33.424922 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:33.425852 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:33.425860 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:33.426115 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:33.427859 (+ 1744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:04:33.427902 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:04:33.432335 (+ 4433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:04:33.437381 (+ 5046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:04:33.437396 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:04:33.437421 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:04:33.438240 (+ 819us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:04:33.439156 (+ 916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:04:33.439163 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:04:33.439169 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:04:33.439724 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:04:33.440429 (+ 705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:04:33.440441 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:04:33.441085 (+ 644us) client_negotiation.cc:769] Sending connection context
0806 03:04:33.441676 (+ 591us) client_negotiation.cc:240] Negotiation successful
0806 03:04:33.442824 (+ 1148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":927,"thread_start_us":614,"threads_started":1}
I20260806 03:04:33.447719 17670 heartbeater.cc:347] Connected to a master server at 127.31.74.254:45687
I20260806 03:04:33.448094 17670 heartbeater.cc:464] Registering TS with master...
I20260806 03:04:33.449424 17670 heartbeater.cc:511] Master 127.31.74.254:45687 requested a full tablet report, sending...
I20260806 03:04:33.452106 17337 ts_manager.cc:194] Registered new tserver with Master: 642d05cc9727477e8622f27bdff90e07 (127.31.74.194:33433)
I20260806 03:04:33.453061 17337 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.74.194@KRBTEST.COM'} at 127.31.74.194:51791
WARNING: no policy specified for kudu/127.31.74.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.195.keytab.
Entry for principal kudu/127.31.74.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.195.keytab.
I20260806 03:04:33.512917 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.195:0
--local_ip_for_outbound_sockets=127.31.74.195
--webserver_interface=127.31.74.195
--webserver_port=0
--tserver_master_addrs=127.31.74.254:45687
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.195.keytab
--principal=kudu/127.31.74.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:45699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 03:04:33.770640 17532 heartbeater.cc:503] Master 127.31.74.254:45687 was elected leader, sending a full tablet report...
W20260806 03:04:33.940989 17676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:04:33.941309 17676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:04:33.941413 17676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:04:33.949368 17676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:04:33.949519 17676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:04:33.949640 17676 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.195
I20260806 03:04:33.967907 17676 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:45699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.195.keytab
--principal=kudu/127.31.74.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.74.195
--webserver_port=0
--tserver_master_addrs=127.31.74.254:45687
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.195
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:04:33.969590 17676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:04:33.971364 17676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:04:33.988739 17684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:04:33.988894 17682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:04:33.989080 17676 server_base.cc:1034] running on GCE node
W20260806 03:04:33.988883 17681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:04:33.990496 17676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:04:33.991955 17676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:04:33.993218 17676 hybrid_clock.cc:648] HybridClock initialized: now 1785985473993129 us; error 76 us; skew 500 ppm
I20260806 03:04:33.993662 17676 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 03:04:33 dist-test-slave-dhph krb5kdc[17291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985473, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:04:33.998415 17676 init.cc:377] Logged in from keytab as kudu/127.31.74.195@KRBTEST.COM (short username kudu)
I20260806 03:04:34.000780 17676 webserver.cc:460] Webserver started at http://127.31.74.195:38503/ using document root <none> and password file <none>
I20260806 03:04:34.001664 17676 fs_manager.cc:359] Metadata directory not provided
I20260806 03:04:34.002274 17676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:04:34.003037 17676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:04:34.007666 17676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/minicluster-data/ts-2/data/instance:
uuid: "26f847aa53f14638a46c1c32c021550c"
format_stamp: "Formatted at 2026-08-06 03:04:34 on dist-test-slave-dhph"
server_key: "ded89d6777c9c31e472e1bfca20d6826"
server_key_iv: "6ac19c754bdfa82a3d5c1ffa86c79c11"
server_key_version: "encryptionkey@0"
I20260806 03:04:34.008772 17676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/minicluster-data/ts-2/wal/instance:
uuid: "26f847aa53f14638a46c1c32c021550c"
format_stamp: "Formatted at 2026-08-06 03:04:34 on dist-test-slave-dhph"
server_key: "ded89d6777c9c31e472e1bfca20d6826"
server_key_iv: "6ac19c754bdfa82a3d5c1ffa86c79c11"
server_key_version: "encryptionkey@0"
I20260806 03:04:34.015926 17676 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.005s
I20260806 03:04:34.021858 17691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:04:34.024408 17676 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260806 03:04:34.024777 17676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/minicluster-data/ts-2/wal
uuid: "26f847aa53f14638a46c1c32c021550c"
format_stamp: "Formatted at 2026-08-06 03:04:34 on dist-test-slave-dhph"
server_key: "ded89d6777c9c31e472e1bfca20d6826"
server_key_iv: "6ac19c754bdfa82a3d5c1ffa86c79c11"
server_key_version: "encryptionkey@0"
I20260806 03:04:34.025205 17676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:04:34.042923 17676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:04:34.050341 17676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:04:34.050776 17676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:04:34.052039 17676 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:04:34.053630 17676 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:04:34.053750 17676 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:04:34.053910 17676 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:04:34.053992 17676 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:04:34.097784 17676 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.195:39367
I20260806 03:04:34.097836 17807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.195:39367 every 8 connection(s)
I20260806 03:04:34.100302 17676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
I20260806 03:04:34.107290 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 17676
I20260806 03:04:34.107491 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/minicluster-data/ts-2/wal/instance
I20260806 03:04:34.107980 32043 external_mini_cluster.cc:1428] Setting key f4f2b74d5de3e9346d0431d68827420c
Aug 06 03:04:34 dist-test-slave-dhph krb5kdc[17291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985473, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.195@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 03:04:34.130548 17810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:34.104226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.195:52325 (local address 127.31.74.254:45687)
0806 03:04:34.104959 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:34.104968 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:34.105559 (+ 591us) server_negotiation.cc:407] Connection header received
0806 03:04:34.108187 (+ 2628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:34.108193 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:34.108292 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:34.108439 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:04:34.110856 (+ 2417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:34.111907 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:34.113644 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:34.114163 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:34.114286 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:34.119213 (+ 4927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:04:34.119218 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:04:34.119222 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:04:34.119265 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:04:34.122665 (+ 3400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:04:34.124829 (+ 2164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:04:34.124835 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:04:34.124838 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:04:34.125042 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:04:34.125943 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:04:34.125947 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:04:34.125950 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:04:34.126628 (+ 678us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:04:34.127739 (+ 1111us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:34.129465 (+ 1726us) server_negotiation.cc:299] Negotiation successful
0806 03:04:34.129872 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":347,"threads_started":1}
I20260806 03:04:34.130687 17811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:34.104385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:45687 (local address 127.31.74.195:52325)
0806 03:04:34.105365 (+ 980us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:34.105424 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0806 03:04:34.107797 (+ 2373us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:04:34.108829 (+ 1032us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:04:34.108846 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:04:34.109607 (+ 761us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:04:34.110641 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:34.110679 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:34.112097 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:34.112101 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:34.113470 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:34.113484 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:34.114464 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:34.114469 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:34.114786 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:34.115492 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:04:34.115520 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:04:34.118830 (+ 3310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:04:34.122966 (+ 4136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:04:34.122976 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:04:34.122994 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:04:34.124529 (+ 1535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:04:34.125316 (+ 787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:04:34.125321 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:04:34.125325 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:04:34.125604 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:04:34.127634 (+ 2030us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:04:34.127646 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:04:34.128147 (+ 501us) client_negotiation.cc:769] Sending connection context
0806 03:04:34.128656 (+ 509us) client_negotiation.cc:240] Negotiation successful
0806 03:04:34.129153 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":703,"thread_start_us":408,"threads_started":1}
I20260806 03:04:34.132807 17808 heartbeater.cc:347] Connected to a master server at 127.31.74.254:45687
I20260806 03:04:34.133126 17808 heartbeater.cc:464] Registering TS with master...
I20260806 03:04:34.133909 17808 heartbeater.cc:511] Master 127.31.74.254:45687 requested a full tablet report, sending...
I20260806 03:04:34.135959 17337 ts_manager.cc:194] Registered new tserver with Master: 26f847aa53f14638a46c1c32c021550c (127.31.74.195:39367)
I20260806 03:04:34.136878 17337 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.74.195@KRBTEST.COM'} at 127.31.74.195:52325
I20260806 03:04:34.141527 32043 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 03:04:34.167600 17810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:34.151106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54202 (local address 127.31.74.254:45687)
0806 03:04:34.151338 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:34.151343 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:34.151364 (+ 21us) server_negotiation.cc:407] Connection header received
0806 03:04:34.151425 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:34.151429 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:34.151515 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:34.151667 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:04:34.153822 (+ 2155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:34.154574 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:34.156063 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:34.157244 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:34.157404 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:34.160603 (+ 3199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:04:34.160609 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:04:34.160613 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:04:34.160658 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:04:34.164203 (+ 3545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:04:34.165112 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:04:34.165122 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:04:34.165127 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:04:34.165300 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:04:34.165752 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:04:34.165760 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:04:34.165767 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:04:34.166143 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:04:34.166457 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:34.166932 (+ 475us) server_negotiation.cc:299] Negotiation successful
0806 03:04:34.167208 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260806 03:04:34.172712 17337 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54202:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 03:04:34.180084 17337 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 03:04:34.211098 17821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:34.195392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.194:33433 (local address 127.0.0.1:44350)
0806 03:04:34.196827 (+ 1435us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:34.196999 (+ 172us) client_negotiation.cc:174] Beginning negotiation
0806 03:04:34.197356 (+ 357us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:04:34.199054 (+ 1698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:04:34.199062 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:04:34.199091 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:04:34.199730 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:34.199757 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:34.202546 (+ 2789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:34.202552 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:34.206367 (+ 3815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:34.206394 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:34.209330 (+ 2936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:34.209340 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:34.209697 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:34.209781 (+ 84us) client_negotiation.cc:769] Sending connection context
0806 03:04:34.209983 (+ 202us) client_negotiation.cc:240] Negotiation successful
0806 03:04:34.210297 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1058,"thread_start_us":780,"threads_started":1}
I20260806 03:04:34.212528 17824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:34.196512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55158 (local address 127.31.74.193:35259)
0806 03:04:34.197421 (+ 909us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:34.197430 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:34.198071 (+ 641us) server_negotiation.cc:407] Connection header received
0806 03:04:34.198271 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:34.198279 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:34.198532 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:34.198840 (+ 308us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:04:34.200229 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:34.202380 (+ 2151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:34.206794 (+ 4414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:34.208960 (+ 2166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:34.209117 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:34.209416 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:34.211775 (+ 2359us) server_negotiation.cc:299] Negotiation successful
0806 03:04:34.212047 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":736,"thread_start_us":525,"threads_started":1}
I20260806 03:04:34.211249 17822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:34.195724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:35259 (local address 127.0.0.1:55158)
0806 03:04:34.197797 (+ 2073us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:34.197815 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 03:04:34.197932 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:04:34.199140 (+ 1208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:04:34.199148 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:04:34.199190 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:04:34.199967 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:34.199985 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:34.202617 (+ 2632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:34.202626 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:34.206500 (+ 3874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:34.206524 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:34.209724 (+ 3200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:34.209734 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:34.210031 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:34.210085 (+ 54us) client_negotiation.cc:769] Sending connection context
0806 03:04:34.210287 (+ 202us) client_negotiation.cc:240] Negotiation successful
0806 03:04:34.210530 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1920,"mutex_wait_us":1,"thread_start_us":1758,"threads_started":1}
I20260806 03:04:34.212913 17823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:34.195907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44350 (local address 127.31.74.194:33433)
0806 03:04:34.196572 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:34.196583 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:34.197582 (+ 999us) server_negotiation.cc:407] Connection header received
0806 03:04:34.197693 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:34.197700 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:34.198503 (+ 803us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:34.198755 (+ 252us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:04:34.200040 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:34.202340 (+ 2300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:34.206705 (+ 4365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:34.208633 (+ 1928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:34.208830 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:34.209144 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:34.211827 (+ 2683us) server_negotiation.cc:299] Negotiation successful
0806 03:04:34.212025 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":1209,"threads_started":1}
I20260806 03:04:34.218572 17603 tablet_service.cc:1467] Processing CreateTablet for tablet 26381904b8db46cfb770d9ac4cf099da (DEFAULT_TABLE table=test-table [id=9cec0026fc274e3fa90535aa34fc1409]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:04:34.218997 17465 tablet_service.cc:1467] Processing CreateTablet for tablet 26381904b8db46cfb770d9ac4cf099da (DEFAULT_TABLE table=test-table [id=9cec0026fc274e3fa90535aa34fc1409]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:04:34.220167 17603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26381904b8db46cfb770d9ac4cf099da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:04:34.220319 17465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26381904b8db46cfb770d9ac4cf099da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:04:34.224999 17821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:34.197881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.195:39367 (local address 127.0.0.1:58888)
0806 03:04:34.211433 (+ 13552us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:34.211452 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 03:04:34.211587 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:04:34.216340 (+ 4753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:04:34.216346 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:04:34.216397 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:04:34.216784 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:34.216805 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:34.219186 (+ 2381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:34.219192 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:34.221433 (+ 2241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:34.221477 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:34.223994 (+ 2517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:34.224011 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:34.224247 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:34.224287 (+ 40us) client_negotiation.cc:769] Sending connection context
0806 03:04:34.224455 (+ 168us) client_negotiation.cc:240] Negotiation successful
0806 03:04:34.224644 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13451,"thread_start_us":16736,"threads_started":1}
I20260806 03:04:34.225193 17826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:34.198382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58888 (local address 127.31.74.195:39367)
0806 03:04:34.215594 (+ 17212us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:34.215605 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:34.215642 (+ 37us) server_negotiation.cc:407] Connection header received
0806 03:04:34.215764 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:34.215774 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:34.216050 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:34.216272 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:04:34.216993 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:34.218988 (+ 1995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:34.221812 (+ 2824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:34.223573 (+ 1761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:34.223742 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:34.224008 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:34.224531 (+ 523us) server_negotiation.cc:299] Negotiation successful
0806 03:04:34.224815 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16770,"thread_start_us":16598,"threads_started":1}
I20260806 03:04:34.230685 17742 tablet_service.cc:1467] Processing CreateTablet for tablet 26381904b8db46cfb770d9ac4cf099da (DEFAULT_TABLE table=test-table [id=9cec0026fc274e3fa90535aa34fc1409]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:04:34.232477 17742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26381904b8db46cfb770d9ac4cf099da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:04:34.235971 17827 tablet_bootstrap.cc:492] T 26381904b8db46cfb770d9ac4cf099da P 53bd6348a73246cbb09421ff1cd113d4: Bootstrap starting.
I20260806 03:04:34.240338 17827 tablet_bootstrap.cc:654] T 26381904b8db46cfb770d9ac4cf099da P 53bd6348a73246cbb09421ff1cd113d4: Neither blocks nor log segments found. Creating new log.
I20260806 03:04:34.241477 17827 log.cc:824] T 26381904b8db46cfb770d9ac4cf099da P 53bd6348a73246cbb09421ff1cd113d4: Log is configured to *not* fsync() on all Append() calls
I20260806 03:04:34.244742 17827 tablet_bootstrap.cc:492] T 26381904b8db46cfb770d9ac4cf099da P 53bd6348a73246cbb09421ff1cd113d4: No bootstrap required, opened a new log
I20260806 03:04:34.245082 17827 ts_tablet_manager.cc:1397] T 26381904b8db46cfb770d9ac4cf099da P 53bd6348a73246cbb09421ff1cd113d4: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260806 03:04:34.246877 17828 tablet_bootstrap.cc:492] T 26381904b8db46cfb770d9ac4cf099da P 642d05cc9727477e8622f27bdff90e07: Bootstrap starting.
I20260806 03:04:34.251706 17828 tablet_bootstrap.cc:654] T 26381904b8db46cfb770d9ac4cf099da P 642d05cc9727477e8622f27bdff90e07: Neither blocks nor log segments found. Creating new log.
I20260806 03:04:34.252362 17830 tablet_bootstrap.cc:492] T 26381904b8db46cfb770d9ac4cf099da P 26f847aa53f14638a46c1c32c021550c: Bootstrap starting.
I20260806 03:04:34.253115 17828 log.cc:824] T 26381904b8db46cfb770d9ac4cf099da P 642d05cc9727477e8622f27bdff90e07: Log is configured to *not* fsync() on all Append() calls
I20260806 03:04:34.255759 17827 raft_consensus.cc:348] T 26381904b8db46cfb770d9ac4cf099da P 53bd6348a73246cbb09421ff1cd113d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642d05cc9727477e8622f27bdff90e07" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 33433 } } peers { permanent_uuid: "53bd6348a73246cbb09421ff1cd113d4" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 35259 } } peers { permanent_uuid: "26f847aa53f14638a46c1c32c021550c" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 39367 } }
I20260806 03:04:34.256269 17827 raft_consensus.cc:374] T 26381904b8db46cfb770d9ac4cf099da P 53bd6348a73246cbb09421ff1cd113d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:04:34.256383 17827 raft_consensus.cc:729] T 26381904b8db46cfb770d9ac4cf099da P 53bd6348a73246cbb09421ff1cd113d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53bd6348a73246cbb09421ff1cd113d4, State: Initialized, Role: FOLLOWER
I20260806 03:04:34.257076 17827 consensus_queue.cc:260] T 26381904b8db46cfb770d9ac4cf099da P 53bd6348a73246cbb09421ff1cd113d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642d05cc9727477e8622f27bdff90e07" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 33433 } } peers { permanent_uuid: "53bd6348a73246cbb09421ff1cd113d4" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 35259 } } peers { permanent_uuid: "26f847aa53f14638a46c1c32c021550c" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 39367 } }
I20260806 03:04:34.257880 17830 tablet_bootstrap.cc:654] T 26381904b8db46cfb770d9ac4cf099da P 26f847aa53f14638a46c1c32c021550c: Neither blocks nor log segments found. Creating new log.
I20260806 03:04:34.258834 17828 tablet_bootstrap.cc:492] T 26381904b8db46cfb770d9ac4cf099da P 642d05cc9727477e8622f27bdff90e07: No bootstrap required, opened a new log
I20260806 03:04:34.259250 17828 ts_tablet_manager.cc:1397] T 26381904b8db46cfb770d9ac4cf099da P 642d05cc9727477e8622f27bdff90e07: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.000s
I20260806 03:04:34.259418 17827 ts_tablet_manager.cc:1428] T 26381904b8db46cfb770d9ac4cf099da P 53bd6348a73246cbb09421ff1cd113d4: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260806 03:04:34.259441 17830 log.cc:824] T 26381904b8db46cfb770d9ac4cf099da P 26f847aa53f14638a46c1c32c021550c: Log is configured to *not* fsync() on all Append() calls
I20260806 03:04:34.263058 17830 tablet_bootstrap.cc:492] T 26381904b8db46cfb770d9ac4cf099da P 26f847aa53f14638a46c1c32c021550c: No bootstrap required, opened a new log
I20260806 03:04:34.263408 17830 ts_tablet_manager.cc:1397] T 26381904b8db46cfb770d9ac4cf099da P 26f847aa53f14638a46c1c32c021550c: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.000s
I20260806 03:04:34.268570 17828 raft_consensus.cc:348] T 26381904b8db46cfb770d9ac4cf099da P 642d05cc9727477e8622f27bdff90e07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642d05cc9727477e8622f27bdff90e07" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 33433 } } peers { permanent_uuid: "53bd6348a73246cbb09421ff1cd113d4" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 35259 } } peers { permanent_uuid: "26f847aa53f14638a46c1c32c021550c" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 39367 } }
I20260806 03:04:34.269018 17828 raft_consensus.cc:374] T 26381904b8db46cfb770d9ac4cf099da P 642d05cc9727477e8622f27bdff90e07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:04:34.269132 17828 raft_consensus.cc:729] T 26381904b8db46cfb770d9ac4cf099da P 642d05cc9727477e8622f27bdff90e07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 642d05cc9727477e8622f27bdff90e07, State: Initialized, Role: FOLLOWER
I20260806 03:04:34.269774 17828 consensus_queue.cc:260] T 26381904b8db46cfb770d9ac4cf099da P 642d05cc9727477e8622f27bdff90e07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642d05cc9727477e8622f27bdff90e07" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 33433 } } peers { permanent_uuid: "53bd6348a73246cbb09421ff1cd113d4" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 35259 } } peers { permanent_uuid: "26f847aa53f14638a46c1c32c021550c" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 39367 } }
I20260806 03:04:34.271112 17670 heartbeater.cc:503] Master 127.31.74.254:45687 was elected leader, sending a full tablet report...
I20260806 03:04:34.271551 17828 ts_tablet_manager.cc:1428] T 26381904b8db46cfb770d9ac4cf099da P 642d05cc9727477e8622f27bdff90e07: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260806 03:04:34.275285 17830 raft_consensus.cc:348] T 26381904b8db46cfb770d9ac4cf099da P 26f847aa53f14638a46c1c32c021550c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642d05cc9727477e8622f27bdff90e07" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 33433 } } peers { permanent_uuid: "53bd6348a73246cbb09421ff1cd113d4" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 35259 } } peers { permanent_uuid: "26f847aa53f14638a46c1c32c021550c" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 39367 } }
I20260806 03:04:34.276142 17830 raft_consensus.cc:374] T 26381904b8db46cfb770d9ac4cf099da P 26f847aa53f14638a46c1c32c021550c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:04:34.276281 17830 raft_consensus.cc:729] T 26381904b8db46cfb770d9ac4cf099da P 26f847aa53f14638a46c1c32c021550c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26f847aa53f14638a46c1c32c021550c, State: Initialized, Role: FOLLOWER
I20260806 03:04:34.277240 17830 consensus_queue.cc:260] T 26381904b8db46cfb770d9ac4cf099da P 26f847aa53f14638a46c1c32c021550c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642d05cc9727477e8622f27bdff90e07" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 33433 } } peers { permanent_uuid: "53bd6348a73246cbb09421ff1cd113d4" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 35259 } } peers { permanent_uuid: "26f847aa53f14638a46c1c32c021550c" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 39367 } }
I20260806 03:04:34.278789 17808 heartbeater.cc:503] Master 127.31.74.254:45687 was elected leader, sending a full tablet report...
I20260806 03:04:34.279150 17830 ts_tablet_manager.cc:1428] T 26381904b8db46cfb770d9ac4cf099da P 26f847aa53f14638a46c1c32c021550c: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
W20260806 03:04:34.352797 17809 tablet.cc:2369] T 26381904b8db46cfb770d9ac4cf099da P 26f847aa53f14638a46c1c32c021550c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 03:04:34.413816 17671 tablet.cc:2369] T 26381904b8db46cfb770d9ac4cf099da P 642d05cc9727477e8622f27bdff90e07: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 03:04:34.461547 17832 raft_consensus.cc:482] T 26381904b8db46cfb770d9ac4cf099da P 53bd6348a73246cbb09421ff1cd113d4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:04:34.461938 17832 raft_consensus.cc:504] T 26381904b8db46cfb770d9ac4cf099da P 53bd6348a73246cbb09421ff1cd113d4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642d05cc9727477e8622f27bdff90e07" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 33433 } } peers { permanent_uuid: "53bd6348a73246cbb09421ff1cd113d4" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 35259 } } peers { permanent_uuid: "26f847aa53f14638a46c1c32c021550c" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 39367 } }
I20260806 03:04:34.463912 17832 leader_election.cc:288] T 26381904b8db46cfb770d9ac4cf099da P 53bd6348a73246cbb09421ff1cd113d4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 642d05cc9727477e8622f27bdff90e07 (127.31.74.194:33433), 26f847aa53f14638a46c1c32c021550c (127.31.74.195:39367)
I20260806 03:04:34.474709 17823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:34.465123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.193:46669 (local address 127.31.74.194:33433)
0806 03:04:34.465602 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:34.465612 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:34.466021 (+ 409us) server_negotiation.cc:407] Connection header received
0806 03:04:34.466322 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:34.466327 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:34.466404 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:34.466482 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:04:34.467892 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:34.469035 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:34.472110 (+ 3075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:34.473379 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:34.473546 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:34.473627 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:34.474263 (+ 636us) server_negotiation.cc:299] Negotiation successful
0806 03:04:34.474430 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324}
I20260806 03:04:34.474705 17826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:34.465119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.193:49957 (local address 127.31.74.195:39367)
0806 03:04:34.465406 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:34.465414 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:34.466646 (+ 1232us) server_negotiation.cc:407] Connection header received
0806 03:04:34.467002 (+ 356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:34.467006 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:34.467068 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:34.467149 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:04:34.468288 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:34.469325 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:34.472089 (+ 2764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:34.473358 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:34.473553 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:34.473627 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:34.474263 (+ 636us) server_negotiation.cc:299] Negotiation successful
0806 03:04:34.474404 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260806 03:04:34.475003 17837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:34.464508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.194:33433 (local address 127.31.74.193:46669)
0806 03:04:34.465822 (+ 1314us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:34.465861 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 03:04:34.466099 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:04:34.466612 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:04:34.466619 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:04:34.466653 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:04:34.467649 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:34.467670 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:34.469242 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:34.469246 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:34.471877 (+ 2631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:34.471892 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:34.473725 (+ 1833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:34.473733 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:34.473972 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:34.474092 (+ 120us) client_negotiation.cc:769] Sending connection context
0806 03:04:34.474212 (+ 120us) client_negotiation.cc:240] Negotiation successful
0806 03:04:34.474372 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1112,"mutex_wait_us":122,"spinlock_wait_cycles":20224,"thread_start_us":1271,"threads_started":1}
I20260806 03:04:34.475662 17836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:34.464612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.195:39367 (local address 127.31.74.193:49957)
0806 03:04:34.466198 (+ 1586us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:34.466217 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 03:04:34.466788 (+ 571us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:04:34.467354 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:04:34.467362 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:04:34.467564 (+ 202us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:04:34.468029 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:34.468040 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:34.469486 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:34.469492 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:34.471875 (+ 2383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:34.471892 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:34.473783 (+ 1891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:34.473791 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:34.474068 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:34.474105 (+ 37us) client_negotiation.cc:769] Sending connection context
0806 03:04:34.474212 (+ 107us) client_negotiation.cc:240] Negotiation successful
0806 03:04:34.474372 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1447,"mutex_wait_us":124,"spinlock_wait_cycles":57472,"thread_start_us":768,"threads_started":1}
I20260806 03:04:34.476516 17624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26381904b8db46cfb770d9ac4cf099da" candidate_uuid: "53bd6348a73246cbb09421ff1cd113d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "642d05cc9727477e8622f27bdff90e07" is_pre_election: true
I20260806 03:04:34.477301 17624 raft_consensus.cc:2454] T 26381904b8db46cfb770d9ac4cf099da P 642d05cc9727477e8622f27bdff90e07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53bd6348a73246cbb09421ff1cd113d4 in term 0.
I20260806 03:04:34.477335 17762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26381904b8db46cfb770d9ac4cf099da" candidate_uuid: "53bd6348a73246cbb09421ff1cd113d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26f847aa53f14638a46c1c32c021550c" is_pre_election: true
I20260806 03:04:34.477875 17762 raft_consensus.cc:2454] T 26381904b8db46cfb770d9ac4cf099da P 26f847aa53f14638a46c1c32c021550c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53bd6348a73246cbb09421ff1cd113d4 in term 0.
I20260806 03:04:34.478229 17419 leader_election.cc:302] T 26381904b8db46cfb770d9ac4cf099da P 53bd6348a73246cbb09421ff1cd113d4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53bd6348a73246cbb09421ff1cd113d4, 642d05cc9727477e8622f27bdff90e07; no voters:
I20260806 03:04:34.478741 17832 raft_consensus.cc:2781] T 26381904b8db46cfb770d9ac4cf099da P 53bd6348a73246cbb09421ff1cd113d4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:04:34.478942 17832 raft_consensus.cc:482] T 26381904b8db46cfb770d9ac4cf099da P 53bd6348a73246cbb09421ff1cd113d4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:04:34.479059 17832 raft_consensus.cc:3037] T 26381904b8db46cfb770d9ac4cf099da P 53bd6348a73246cbb09421ff1cd113d4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:04:34.481578 17832 raft_consensus.cc:504] T 26381904b8db46cfb770d9ac4cf099da P 53bd6348a73246cbb09421ff1cd113d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642d05cc9727477e8622f27bdff90e07" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 33433 } } peers { permanent_uuid: "53bd6348a73246cbb09421ff1cd113d4" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 35259 } } peers { permanent_uuid: "26f847aa53f14638a46c1c32c021550c" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 39367 } }
I20260806 03:04:34.482375 17832 leader_election.cc:288] T 26381904b8db46cfb770d9ac4cf099da P 53bd6348a73246cbb09421ff1cd113d4 [CANDIDATE]: Term 1 election: Requested vote from peers 642d05cc9727477e8622f27bdff90e07 (127.31.74.194:33433), 26f847aa53f14638a46c1c32c021550c (127.31.74.195:39367)
I20260806 03:04:34.482969 17624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26381904b8db46cfb770d9ac4cf099da" candidate_uuid: "53bd6348a73246cbb09421ff1cd113d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "642d05cc9727477e8622f27bdff90e07"
I20260806 03:04:34.483031 17762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26381904b8db46cfb770d9ac4cf099da" candidate_uuid: "53bd6348a73246cbb09421ff1cd113d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26f847aa53f14638a46c1c32c021550c"
I20260806 03:04:34.483222 17624 raft_consensus.cc:3037] T 26381904b8db46cfb770d9ac4cf099da P 642d05cc9727477e8622f27bdff90e07 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:04:34.483270 17762 raft_consensus.cc:3037] T 26381904b8db46cfb770d9ac4cf099da P 26f847aa53f14638a46c1c32c021550c [term 0 FOLLOWER]: Advancing to term 1
W20260806 03:04:34.483827 17533 tablet.cc:2369] T 26381904b8db46cfb770d9ac4cf099da P 53bd6348a73246cbb09421ff1cd113d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 03:04:34.485795 17624 raft_consensus.cc:2454] T 26381904b8db46cfb770d9ac4cf099da P 642d05cc9727477e8622f27bdff90e07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53bd6348a73246cbb09421ff1cd113d4 in term 1.
I20260806 03:04:34.485955 17762 raft_consensus.cc:2454] T 26381904b8db46cfb770d9ac4cf099da P 26f847aa53f14638a46c1c32c021550c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53bd6348a73246cbb09421ff1cd113d4 in term 1.
I20260806 03:04:34.486753 17419 leader_election.cc:302] T 26381904b8db46cfb770d9ac4cf099da P 53bd6348a73246cbb09421ff1cd113d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53bd6348a73246cbb09421ff1cd113d4, 642d05cc9727477e8622f27bdff90e07; no voters:
I20260806 03:04:34.487167 17832 raft_consensus.cc:2781] T 26381904b8db46cfb770d9ac4cf099da P 53bd6348a73246cbb09421ff1cd113d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:04:34.487813 17832 raft_consensus.cc:686] T 26381904b8db46cfb770d9ac4cf099da P 53bd6348a73246cbb09421ff1cd113d4 [term 1 LEADER]: Becoming Leader. State: Replica: 53bd6348a73246cbb09421ff1cd113d4, State: Running, Role: LEADER
I20260806 03:04:34.488466 17832 consensus_queue.cc:237] T 26381904b8db46cfb770d9ac4cf099da P 53bd6348a73246cbb09421ff1cd113d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642d05cc9727477e8622f27bdff90e07" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 33433 } } peers { permanent_uuid: "53bd6348a73246cbb09421ff1cd113d4" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 35259 } } peers { permanent_uuid: "26f847aa53f14638a46c1c32c021550c" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 39367 } }
I20260806 03:04:34.493443 17336 catalog_manager.cc:5495] T 26381904b8db46cfb770d9ac4cf099da P 53bd6348a73246cbb09421ff1cd113d4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 53bd6348a73246cbb09421ff1cd113d4 (127.31.74.193). New cstate: current_term: 1 leader_uuid: "53bd6348a73246cbb09421ff1cd113d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642d05cc9727477e8622f27bdff90e07" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 33433 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53bd6348a73246cbb09421ff1cd113d4" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 35259 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26f847aa53f14638a46c1c32c021550c" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 39367 } health_report { overall_health: UNKNOWN } } }
I20260806 03:04:34.515656 17824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:34.508701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55166 (local address 127.31.74.193:35259)
0806 03:04:34.508940 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:34.508944 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:34.508964 (+ 20us) server_negotiation.cc:407] Connection header received
0806 03:04:34.509214 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:34.509219 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:34.509350 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:34.509506 (+ 156us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 03:04:34.509996 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:34.510963 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:34.512892 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:34.513375 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:34.513504 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:34.514126 (+ 622us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 03:04:34.514818 (+ 692us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 03:04:34.514950 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:34.515173 (+ 223us) server_negotiation.cc:299] Negotiation successful
0806 03:04:34.515258 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260806 03:04:34.527352 17762 raft_consensus.cc:1261] T 26381904b8db46cfb770d9ac4cf099da P 26f847aa53f14638a46c1c32c021550c [term 1 FOLLOWER]: Refusing update from remote peer 53bd6348a73246cbb09421ff1cd113d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:04:34.527352 17624 raft_consensus.cc:1261] T 26381904b8db46cfb770d9ac4cf099da P 642d05cc9727477e8622f27bdff90e07 [term 1 FOLLOWER]: Refusing update from remote peer 53bd6348a73246cbb09421ff1cd113d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:04:34.528719 17838 consensus_queue.cc:1035] T 26381904b8db46cfb770d9ac4cf099da P 53bd6348a73246cbb09421ff1cd113d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "642d05cc9727477e8622f27bdff90e07" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 33433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:04:34.529404 17832 consensus_queue.cc:1035] T 26381904b8db46cfb770d9ac4cf099da P 53bd6348a73246cbb09421ff1cd113d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26f847aa53f14638a46c1c32c021550c" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 39367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:04:34.574806 17336 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 03:04:34.575145 17336 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 03:04:34.580302 17336 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ab14a05097ef4c97b8636d17618f988a: Sending DeleteTablet for 3 replicas of tablet 26381904b8db46cfb770d9ac4cf099da
I20260806 03:04:34.582998 17603 tablet_service.cc:1514] Processing DeleteTablet for tablet 26381904b8db46cfb770d9ac4cf099da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:04:34 UTC) from {username='kudu', principal='kudu/127.31.74.254@KRBTEST.COM'} at 127.0.0.1:44350
I20260806 03:04:34.583210 17742 tablet_service.cc:1514] Processing DeleteTablet for tablet 26381904b8db46cfb770d9ac4cf099da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:04:34 UTC) from {username='kudu', principal='kudu/127.31.74.254@KRBTEST.COM'} at 127.0.0.1:58888
I20260806 03:04:34.584683 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 17401
I20260806 03:04:34.592900 17465 tablet_service.cc:1514] Processing DeleteTablet for tablet 26381904b8db46cfb770d9ac4cf099da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:04:34 UTC) from {username='kudu', principal='kudu/127.31.74.254@KRBTEST.COM'} at 127.0.0.1:55158
I20260806 03:04:34.594396 17850 tablet_replica.cc:330] T 26381904b8db46cfb770d9ac4cf099da P 642d05cc9727477e8622f27bdff90e07: stopping tablet replica
I20260806 03:04:34.594379 17851 tablet_replica.cc:330] T 26381904b8db46cfb770d9ac4cf099da P 26f847aa53f14638a46c1c32c021550c: stopping tablet replica
I20260806 03:04:34.594960 17850 raft_consensus.cc:2229] T 26381904b8db46cfb770d9ac4cf099da P 642d05cc9727477e8622f27bdff90e07 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:04:34.595636 17850 raft_consensus.cc:2258] T 26381904b8db46cfb770d9ac4cf099da P 642d05cc9727477e8622f27bdff90e07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:04:34.598400 17850 ts_tablet_manager.cc:1905] T 26381904b8db46cfb770d9ac4cf099da P 642d05cc9727477e8622f27bdff90e07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:04:34.599892 17851 raft_consensus.cc:2229] T 26381904b8db46cfb770d9ac4cf099da P 26f847aa53f14638a46c1c32c021550c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:04:34.600513 17851 raft_consensus.cc:2258] T 26381904b8db46cfb770d9ac4cf099da P 26f847aa53f14638a46c1c32c021550c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:04:34.608382 17850 ts_tablet_manager.cc:1918] T 26381904b8db46cfb770d9ac4cf099da P 642d05cc9727477e8622f27bdff90e07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 03:04:34.608641 17850 log.cc:1196] T 26381904b8db46cfb770d9ac4cf099da P 642d05cc9727477e8622f27bdff90e07: Deleting WAL directory at /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/minicluster-data/ts-1/wal/wals/26381904b8db46cfb770d9ac4cf099da
I20260806 03:04:34.609375 17850 ts_tablet_manager.cc:1939] T 26381904b8db46cfb770d9ac4cf099da P 642d05cc9727477e8622f27bdff90e07: Deleting consensus metadata
I20260806 03:04:34.611375 17854 tablet_replica.cc:330] T 26381904b8db46cfb770d9ac4cf099da P 53bd6348a73246cbb09421ff1cd113d4: stopping tablet replica
I20260806 03:04:34.611945 17854 raft_consensus.cc:2229] T 26381904b8db46cfb770d9ac4cf099da P 53bd6348a73246cbb09421ff1cd113d4 [term 1 LEADER]: Raft consensus shutting down.
I20260806 03:04:34.612097 17323 catalog_manager.cc:4841] TS 642d05cc9727477e8622f27bdff90e07 (127.31.74.194:33433): tablet 26381904b8db46cfb770d9ac4cf099da (table test-table [id=9cec0026fc274e3fa90535aa34fc1409]) successfully deleted
I20260806 03:04:34.612757 17854 raft_consensus.cc:2258] T 26381904b8db46cfb770d9ac4cf099da P 53bd6348a73246cbb09421ff1cd113d4 [term 1 FOLLOWER]: Raft consensus is shut down!
Aug 06 03:04:34 dist-test-slave-dhph krb5kdc[17291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.74.193@KRBTEST.COM
I20260806 03:04:34.615162 17851 ts_tablet_manager.cc:1905] T 26381904b8db46cfb770d9ac4cf099da P 26f847aa53f14638a46c1c32c021550c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:04:34.621255 17854 ts_tablet_manager.cc:1905] T 26381904b8db46cfb770d9ac4cf099da P 53bd6348a73246cbb09421ff1cd113d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:04:34.633621 17854 ts_tablet_manager.cc:1918] T 26381904b8db46cfb770d9ac4cf099da P 53bd6348a73246cbb09421ff1cd113d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 03:04:34.633817 17854 log.cc:1196] T 26381904b8db46cfb770d9ac4cf099da P 53bd6348a73246cbb09421ff1cd113d4: Deleting WAL directory at /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/minicluster-data/ts-0/wal/wals/26381904b8db46cfb770d9ac4cf099da
I20260806 03:04:34.634339 17854 ts_tablet_manager.cc:1939] T 26381904b8db46cfb770d9ac4cf099da P 53bd6348a73246cbb09421ff1cd113d4: Deleting consensus metadata
I20260806 03:04:34.634733 17851 ts_tablet_manager.cc:1918] T 26381904b8db46cfb770d9ac4cf099da P 26f847aa53f14638a46c1c32c021550c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 03:04:34.634966 17851 log.cc:1196] T 26381904b8db46cfb770d9ac4cf099da P 26f847aa53f14638a46c1c32c021550c: Deleting WAL directory at /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785985105255475-32043-0/minicluster-data/ts-2/wal/wals/26381904b8db46cfb770d9ac4cf099da
I20260806 03:04:34.635710 17851 ts_tablet_manager.cc:1939] T 26381904b8db46cfb770d9ac4cf099da P 26f847aa53f14638a46c1c32c021550c: Deleting consensus metadata
I20260806 03:04:34.636981 17322 catalog_manager.cc:4841] TS 53bd6348a73246cbb09421ff1cd113d4 (127.31.74.193:35259): tablet 26381904b8db46cfb770d9ac4cf099da (table test-table [id=9cec0026fc274e3fa90535aa34fc1409]) successfully deleted
I20260806 03:04:34.638127 17322 catalog_manager.cc:4841] TS 26f847aa53f14638a46c1c32c021550c (127.31.74.195:39367): tablet 26381904b8db46cfb770d9ac4cf099da (table test-table [id=9cec0026fc274e3fa90535aa34fc1409]) successfully deleted
I20260806 03:04:34.639070 17824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:34.589498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55182 (local address 127.31.74.193:35259)
0806 03:04:34.589815 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:34.589826 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:34.589862 (+ 36us) server_negotiation.cc:407] Connection header received
0806 03:04:34.589974 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:34.589985 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:34.593668 (+ 3683us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:34.593935 (+ 267us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:04:34.600614 (+ 6679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:34.602370 (+ 1756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:34.605252 (+ 2882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:34.606647 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:34.606809 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:34.616232 (+ 9423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:04:34.616252 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:04:34.616271 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:04:34.616324 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:04:34.625397 (+ 9073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:04:34.628619 (+ 3222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:04:34.628635 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:04:34.628640 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:04:34.629267 (+ 627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:04:34.631357 (+ 2090us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:04:34.631364 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:04:34.631368 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:04:34.632020 (+ 652us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:04:34.632444 (+ 424us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:34.638255 (+ 5811us) server_negotiation.cc:299] Negotiation successful
0806 03:04:34.638619 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260806 03:04:34.641142 17527 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 03:04:34.764514 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 17401
I20260806 03:04:34.789234 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 17538
Aug 06 03:04:34 dist-test-slave-dhph krb5kdc[17291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.74.194@KRBTEST.COM
I20260806 03:04:34.810283 17823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:34.790117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44352 (local address 127.31.74.194:33433)
0806 03:04:34.790317 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:34.790322 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:34.790343 (+ 21us) server_negotiation.cc:407] Connection header received
0806 03:04:34.790548 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:34.790553 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:34.790739 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:34.790847 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:04:34.792738 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:34.793956 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:34.796076 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:34.796553 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:34.796757 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:34.803151 (+ 6394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:04:34.803171 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:04:34.803190 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:04:34.803242 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:04:34.807029 (+ 3787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:04:34.807939 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:04:34.807950 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:04:34.807956 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:04:34.808110 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:04:34.808561 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:04:34.808566 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:04:34.808571 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:04:34.808993 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:04:34.809310 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:34.809648 (+ 338us) server_negotiation.cc:299] Negotiation successful
0806 03:04:34.809905 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 03:04:34.811311 17665 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 03:04:34.915864 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 17538
I20260806 03:04:34.941076 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 17676
Aug 06 03:04:34 dist-test-slave-dhph krb5kdc[17291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.74.195@KRBTEST.COM
I20260806 03:04:34.965983 17826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:34.942277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58900 (local address 127.31.74.195:39367)
0806 03:04:34.942618 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:34.942625 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:34.942650 (+ 25us) server_negotiation.cc:407] Connection header received
0806 03:04:34.942921 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:34.942928 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:34.943009 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:34.943155 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:04:34.944654 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:34.945468 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:34.948158 (+ 2690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:34.949201 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:34.949643 (+ 442us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:34.957364 (+ 7721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:04:34.957384 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:04:34.957403 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:04:34.957451 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:04:34.961605 (+ 4154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:04:34.962562 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:04:34.962570 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:04:34.962572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:04:34.963071 (+ 499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:04:34.963845 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:04:34.963858 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:04:34.963861 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:04:34.964290 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:04:34.964777 (+ 487us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:34.965066 (+ 289us) server_negotiation.cc:299] Negotiation successful
0806 03:04:34.965490 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162}
I20260806 03:04:34.966903 17803 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 03:04:35.075493 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 17676
I20260806 03:04:35.095677 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 17305
I20260806 03:04:35.097038 17370 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 03:04:35.233845 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 17305
2026-08-06T03:04:35Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (5495 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-06T03:04:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:04:35Z Disabled control of system clock
I20260806 03:04:35.357546 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785985105255475-32043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785985105255475-32043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785985105255475-32043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:37897
--webserver_interface=127.31.74.254
--webserver_port=0
--builtin_ntp_servers=127.31.74.212:44409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.74.254:37897
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 03:04:35.806936 17862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:04:35.807299 17862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:04:35.807396 17862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:04:35.816123 17862 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:04:35.816288 17862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:04:35.816334 17862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:04:35.816471 17862 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:04:35.816538 17862 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:04:35.833334 17862 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:44409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785985105255475-32043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785985105255475-32043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.74.254:37897
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:37897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.74.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785985105255475-32043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:04:35.835911 17862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:04:35.840993 17862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:04:35.861178 17870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:04:35.861356 17868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:04:35.861420 17867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:04:35.862262 17862 server_base.cc:1034] running on GCE node
I20260806 03:04:35.863492 17862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:04:35.865401 17862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:04:35.867311 17862 hybrid_clock.cc:648] HybridClock initialized: now 1785985475866979 us; error 177 us; skew 500 ppm
I20260806 03:04:35.867996 17862 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 03:04:35.872190 17862 webserver.cc:460] Webserver started at http://127.31.74.254:35427/ using document root <none> and password file <none>
I20260806 03:04:35.873186 17862 fs_manager.cc:359] Metadata directory not provided
I20260806 03:04:35.873690 17862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:04:35.875144 17862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:04:35.882655 17862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785985105255475-32043-0/minicluster-data/master-0/data/instance:
uuid: "e976b933af1443328d21321228453ed1"
format_stamp: "Formatted at 2026-08-06 03:04:35 on dist-test-slave-dhph"
server_key: "83d66fec521d9b0d3fd9d286423ecf15"
server_key_iv: "c1760d8aa0148363656eb35026c5e642"
server_key_version: "encryptionkey@0"
I20260806 03:04:35.883873 17862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785985105255475-32043-0/minicluster-data/master-0/wal/instance:
uuid: "e976b933af1443328d21321228453ed1"
format_stamp: "Formatted at 2026-08-06 03:04:35 on dist-test-slave-dhph"
server_key: "83d66fec521d9b0d3fd9d286423ecf15"
server_key_iv: "c1760d8aa0148363656eb35026c5e642"
server_key_version: "encryptionkey@0"
I20260806 03:04:35.893117 17862 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.009s sys 0.000s
I20260806 03:04:35.898495 17876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:04:35.900656 17862 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.002s
I20260806 03:04:35.901062 17862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785985105255475-32043-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785985105255475-32043-0/minicluster-data/master-0/wal
uuid: "e976b933af1443328d21321228453ed1"
format_stamp: "Formatted at 2026-08-06 03:04:35 on dist-test-slave-dhph"
server_key: "83d66fec521d9b0d3fd9d286423ecf15"
server_key_iv: "c1760d8aa0148363656eb35026c5e642"
server_key_version: "encryptionkey@0"
I20260806 03:04:35.901409 17862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785985105255475-32043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785985105255475-32043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785985105255475-32043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:04:35.931062 17862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:04:35.932933 17862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:04:35.933954 17862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:04:35.965878 17862 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.254:37897
I20260806 03:04:35.965937 17929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.254:37897 every 8 connection(s)
I20260806 03:04:35.967670 17862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
I20260806 03:04:35.972416 17930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:04:35.976490 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 17862
I20260806 03:04:35.976761 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785985105255475-32043-0/minicluster-data/master-0/wal/instance
I20260806 03:04:35.977294 32043 external_mini_cluster.cc:1428] Setting key a9fc45c67837b12715f3f8ac6814e53f
I20260806 03:04:35.988286 17930 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e976b933af1443328d21321228453ed1: Bootstrap starting.
I20260806 03:04:35.996572 17930 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e976b933af1443328d21321228453ed1: Neither blocks nor log segments found. Creating new log.
I20260806 03:04:35.998171 17930 log.cc:824] T 00000000000000000000000000000000 P e976b933af1443328d21321228453ed1: Log is configured to *not* fsync() on all Append() calls
I20260806 03:04:36.001467 17933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:35.979881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38856 (local address 127.31.74.254:37897)
0806 03:04:35.981370 (+ 1489us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:35.981383 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:35.981434 (+ 51us) server_negotiation.cc:407] Connection header received
0806 03:04:35.983532 (+ 2098us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:35.983561 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:35.984151 (+ 590us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:35.984661 (+ 510us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:04:35.985382 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:35.987112 (+ 1730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:35.990007 (+ 2895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:35.995262 (+ 5255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:35.995588 (+ 326us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:35.997557 (+ 1969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:04:35.997567 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:04:35.997584 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:04:35.998225 (+ 641us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:04:35.998333 (+ 108us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:04:35.998404 (+ 71us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:04:35.998537 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:35.999057 (+ 520us) server_negotiation.cc:299] Negotiation successful
0806 03:04:35.999508 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":788,"thread_start_us":419,"threads_started":1}
I20260806 03:04:36.003321 17930 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e976b933af1443328d21321228453ed1: No bootstrap required, opened a new log
I20260806 03:04:36.013654 17930 raft_consensus.cc:348] T 00000000000000000000000000000000 P e976b933af1443328d21321228453ed1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e976b933af1443328d21321228453ed1" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 37897 } }
I20260806 03:04:36.014178 17930 raft_consensus.cc:374] T 00000000000000000000000000000000 P e976b933af1443328d21321228453ed1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:04:36.014314 17930 raft_consensus.cc:729] T 00000000000000000000000000000000 P e976b933af1443328d21321228453ed1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e976b933af1443328d21321228453ed1, State: Initialized, Role: FOLLOWER
I20260806 03:04:36.015074 17930 consensus_queue.cc:260] T 00000000000000000000000000000000 P e976b933af1443328d21321228453ed1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e976b933af1443328d21321228453ed1" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 37897 } }
I20260806 03:04:36.015441 17930 raft_consensus.cc:388] T 00000000000000000000000000000000 P e976b933af1443328d21321228453ed1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:04:36.015641 17930 raft_consensus.cc:482] T 00000000000000000000000000000000 P e976b933af1443328d21321228453ed1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:04:36.015861 17930 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e976b933af1443328d21321228453ed1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:04:36.019058 17930 raft_consensus.cc:504] T 00000000000000000000000000000000 P e976b933af1443328d21321228453ed1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e976b933af1443328d21321228453ed1" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 37897 } }
I20260806 03:04:36.019695 17930 leader_election.cc:302] T 00000000000000000000000000000000 P e976b933af1443328d21321228453ed1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e976b933af1443328d21321228453ed1; no voters:
I20260806 03:04:36.020704 17930 leader_election.cc:288] T 00000000000000000000000000000000 P e976b933af1443328d21321228453ed1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:04:36.020922 17935 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e976b933af1443328d21321228453ed1 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:04:36.022186 17935 raft_consensus.cc:686] T 00000000000000000000000000000000 P e976b933af1443328d21321228453ed1 [term 1 LEADER]: Becoming Leader. State: Replica: e976b933af1443328d21321228453ed1, State: Running, Role: LEADER
I20260806 03:04:36.022822 17935 consensus_queue.cc:237] T 00000000000000000000000000000000 P e976b933af1443328d21321228453ed1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e976b933af1443328d21321228453ed1" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 37897 } }
I20260806 03:04:36.024183 17930 sys_catalog.cc:564] T 00000000000000000000000000000000 P e976b933af1443328d21321228453ed1 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:04:36.027170 17936 sys_catalog.cc:455] T 00000000000000000000000000000000 P e976b933af1443328d21321228453ed1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e976b933af1443328d21321228453ed1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e976b933af1443328d21321228453ed1" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 37897 } } }
I20260806 03:04:36.027413 17937 sys_catalog.cc:455] T 00000000000000000000000000000000 P e976b933af1443328d21321228453ed1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e976b933af1443328d21321228453ed1. Latest consensus state: current_term: 1 leader_uuid: "e976b933af1443328d21321228453ed1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e976b933af1443328d21321228453ed1" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 37897 } } }
I20260806 03:04:36.028882 17937 sys_catalog.cc:458] T 00000000000000000000000000000000 P e976b933af1443328d21321228453ed1 [sys.catalog]: This master's current role is: LEADER
I20260806 03:04:36.029618 17936 sys_catalog.cc:458] T 00000000000000000000000000000000 P e976b933af1443328d21321228453ed1 [sys.catalog]: This master's current role is: LEADER
I20260806 03:04:36.031190 17944 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:04:36.038995 17944 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:04:36.052667 17944 catalog_manager.cc:1347] Generated new cluster ID: 60786029b4244d109e0bf66bd423356c
I20260806 03:04:36.052825 17944 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:04:36.070719 17944 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:04:36.073757 17944 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:04:36.082953 17944 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e976b933af1443328d21321228453ed1: Generated new TSK 0
I20260806 03:04:36.084388 17944 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 03:04:36.094081 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785985105255475-32043-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785985105255475-32043-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785985105255475-32043-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.193:0
--local_ip_for_outbound_sockets=127.31.74.193
--webserver_interface=127.31.74.193
--webserver_port=0
--tserver_master_addrs=127.31.74.254:37897
--builtin_ntp_servers=127.31.74.212:44409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 03:04:36.540026 17955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:04:36.540390 17955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:04:36.540496 17955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:04:36.550504 17955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:04:36.550732 17955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:04:36.550868 17955 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.193
I20260806 03:04:36.562274 17955 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:44409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785985105255475-32043-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785985105255475-32043-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.74.193
--webserver_port=0
--tserver_master_addrs=127.31.74.254:37897
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.193
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785985105255475-32043-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:04:36.563846 17955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:04:36.566190 17955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:04:36.585938 17961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:04:36.586963 17963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:04:36.586916 17960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:04:36.589174 17955 server_base.cc:1034] running on GCE node
I20260806 03:04:36.590008 17955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:04:36.591483 17955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:04:36.592797 17955 hybrid_clock.cc:648] HybridClock initialized: now 1785985476592736 us; error 46 us; skew 500 ppm
I20260806 03:04:36.593251 17955 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 03:04:36.597254 17955 webserver.cc:460] Webserver started at http://127.31.74.193:38581/ using document root <none> and password file <none>
I20260806 03:04:36.598512 17955 fs_manager.cc:359] Metadata directory not provided
I20260806 03:04:36.598717 17955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:04:36.599354 17955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:04:36.604838 17955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785985105255475-32043-0/minicluster-data/ts-0/data/instance:
uuid: "55edf67c2f004e809fc591077349866b"
format_stamp: "Formatted at 2026-08-06 03:04:36 on dist-test-slave-dhph"
server_key: "e48e4ad8cfb5d69006062eb40f94cd28"
server_key_iv: "82104a8cf8a5f028a802575fbcbbfdae"
server_key_version: "encryptionkey@0"
I20260806 03:04:36.606242 17955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785985105255475-32043-0/minicluster-data/ts-0/wal/instance:
uuid: "55edf67c2f004e809fc591077349866b"
format_stamp: "Formatted at 2026-08-06 03:04:36 on dist-test-slave-dhph"
server_key: "e48e4ad8cfb5d69006062eb40f94cd28"
server_key_iv: "82104a8cf8a5f028a802575fbcbbfdae"
server_key_version: "encryptionkey@0"
I20260806 03:04:36.612968 17955 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260806 03:04:36.618968 17969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:04:36.622290 17955 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.002s
I20260806 03:04:36.622581 17955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785985105255475-32043-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785985105255475-32043-0/minicluster-data/ts-0/wal
uuid: "55edf67c2f004e809fc591077349866b"
format_stamp: "Formatted at 2026-08-06 03:04:36 on dist-test-slave-dhph"
server_key: "e48e4ad8cfb5d69006062eb40f94cd28"
server_key_iv: "82104a8cf8a5f028a802575fbcbbfdae"
server_key_version: "encryptionkey@0"
I20260806 03:04:36.622903 17955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785985105255475-32043-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785985105255475-32043-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785985105255475-32043-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:04:36.646369 17955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:04:36.647761 17955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:04:36.648169 17955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:04:36.649410 17955 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:04:36.652329 17955 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:04:36.652446 17955 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:04:36.652582 17955 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:04:36.652742 17955 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:04:36.702468 17955 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.193:35439
I20260806 03:04:36.702564 18083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.193:35439 every 8 connection(s)
I20260806 03:04:36.703951 17955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
I20260806 03:04:36.712845 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 17955
I20260806 03:04:36.713073 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785985105255475-32043-0/minicluster-data/ts-0/wal/instance
I20260806 03:04:36.713620 32043 external_mini_cluster.cc:1428] Setting key cea460f2e59ffcba2c2c049e25bee702
I20260806 03:04:36.718910 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785985105255475-32043-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785985105255475-32043-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785985105255475-32043-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.194:0
--local_ip_for_outbound_sockets=127.31.74.194
--webserver_interface=127.31.74.194
--webserver_port=0
--tserver_master_addrs=127.31.74.254:37897
--builtin_ntp_servers=127.31.74.212:44409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 03:04:36.724344 18087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:36.707995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.193:47765 (local address 127.31.74.254:37897)
0806 03:04:36.708801 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:36.708807 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:36.710785 (+ 1978us) server_negotiation.cc:407] Connection header received
0806 03:04:36.713732 (+ 2947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:36.713740 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:36.713861 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:36.714024 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:04:36.715627 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:36.716871 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:36.719828 (+ 2957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:36.721181 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:36.721440 (+ 259us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:36.723311 (+ 1871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:04:36.723317 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:04:36.723321 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:04:36.723345 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:04:36.723392 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:04:36.723406 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:04:36.723522 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:36.723799 (+ 277us) server_negotiation.cc:299] Negotiation successful
0806 03:04:36.723926 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":368,"threads_started":1}
I20260806 03:04:36.725659 18088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:36.709043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:37897 (local address 127.31.74.193:47765)
0806 03:04:36.710461 (+ 1418us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:36.710515 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 03:04:36.713290 (+ 2775us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:04:36.714363 (+ 1073us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:04:36.714377 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:04:36.714563 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:04:36.715391 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:36.715418 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:36.717086 (+ 1668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:36.717093 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:36.719317 (+ 2224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:36.719335 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:36.721577 (+ 2242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:36.721588 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:36.721903 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:36.722859 (+ 956us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 03:04:36.722862 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:04:36.722909 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 03:04:36.723013 (+ 104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:04:36.723522 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:04:36.723537 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:04:36.723550 (+ 13us) client_negotiation.cc:769] Sending connection context
0806 03:04:36.723774 (+ 224us) client_negotiation.cc:240] Negotiation successful
0806 03:04:36.724275 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":739,"thread_start_us":489,"threads_started":1}
I20260806 03:04:36.728245 18084 heartbeater.cc:347] Connected to a master server at 127.31.74.254:37897
I20260806 03:04:36.728658 18084 heartbeater.cc:464] Registering TS with master...
I20260806 03:04:36.729918 18084 heartbeater.cc:511] Master 127.31.74.254:37897 requested a full tablet report, sending...
I20260806 03:04:36.733032 17894 ts_manager.cc:194] Registered new tserver with Master: 55edf67c2f004e809fc591077349866b (127.31.74.193:35439)
I20260806 03:04:36.734891 17894 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.74.193:47765
W20260806 03:04:37.169795 18089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:04:37.170121 18089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:04:37.170269 18089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:04:37.179037 18089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:04:37.179171 18089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:04:37.179368 18089 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.194
I20260806 03:04:37.190918 18089 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:44409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785985105255475-32043-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785985105255475-32043-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.74.194
--webserver_port=0
--tserver_master_addrs=127.31.74.254:37897
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.194
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785985105255475-32043-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:04:37.192783 18089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:04:37.194685 18089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:04:37.211639 18094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:04:37.211900 18095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:04:37.212502 18097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:04:37.213155 18089 server_base.cc:1034] running on GCE node
I20260806 03:04:37.213917 18089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:04:37.215112 18089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:04:37.216567 18089 hybrid_clock.cc:648] HybridClock initialized: now 1785985477216532 us; error 74 us; skew 500 ppm
I20260806 03:04:37.217015 18089 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 03:04:37.219807 18089 webserver.cc:460] Webserver started at http://127.31.74.194:33493/ using document root <none> and password file <none>
I20260806 03:04:37.220574 18089 fs_manager.cc:359] Metadata directory not provided
I20260806 03:04:37.220674 18089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:04:37.221009 18089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:04:37.225243 18089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785985105255475-32043-0/minicluster-data/ts-1/data/instance:
uuid: "0fbdeac3ff794b62abf26137d6d1076b"
format_stamp: "Formatted at 2026-08-06 03:04:37 on dist-test-slave-dhph"
server_key: "7f85f1612f01e7d8fa7acbba73301d40"
server_key_iv: "870f90bba77c2e95401fc1a8c70e10cd"
server_key_version: "encryptionkey@0"
I20260806 03:04:37.226537 18089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785985105255475-32043-0/minicluster-data/ts-1/wal/instance:
uuid: "0fbdeac3ff794b62abf26137d6d1076b"
format_stamp: "Formatted at 2026-08-06 03:04:37 on dist-test-slave-dhph"
server_key: "7f85f1612f01e7d8fa7acbba73301d40"
server_key_iv: "870f90bba77c2e95401fc1a8c70e10cd"
server_key_version: "encryptionkey@0"
I20260806 03:04:37.234098 18089 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260806 03:04:37.239363 18103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:04:37.241307 18089 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260806 03:04:37.241539 18089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785985105255475-32043-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785985105255475-32043-0/minicluster-data/ts-1/wal
uuid: "0fbdeac3ff794b62abf26137d6d1076b"
format_stamp: "Formatted at 2026-08-06 03:04:37 on dist-test-slave-dhph"
server_key: "7f85f1612f01e7d8fa7acbba73301d40"
server_key_iv: "870f90bba77c2e95401fc1a8c70e10cd"
server_key_version: "encryptionkey@0"
I20260806 03:04:37.241811 18089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785985105255475-32043-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785985105255475-32043-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785985105255475-32043-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:04:37.263942 18089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:04:37.264979 18089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:04:37.265388 18089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:04:37.267148 18089 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:04:37.269016 18089 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:04:37.269122 18089 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:04:37.269272 18089 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:04:37.269421 18089 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:04:37.316380 18089 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.194:35477
I20260806 03:04:37.316434 18220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.194:35477 every 8 connection(s)
I20260806 03:04:37.317992 18089 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
I20260806 03:04:37.327775 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 18089
I20260806 03:04:37.328016 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785985105255475-32043-0/minicluster-data/ts-1/wal/instance
I20260806 03:04:37.328586 32043 external_mini_cluster.cc:1428] Setting key 55afdb4b052bcdf2d050e190591a376a
I20260806 03:04:37.332293 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785985105255475-32043-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785985105255475-32043-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785985105255475-32043-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.195:0
--local_ip_for_outbound_sockets=127.31.74.195
--webserver_interface=127.31.74.195
--webserver_port=0
--tserver_master_addrs=127.31.74.254:37897
--builtin_ntp_servers=127.31.74.212:44409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 03:04:37.333592 18223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:37.321257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.194:52943 (local address 127.31.74.254:37897)
0806 03:04:37.321843 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:37.321848 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:37.323180 (+ 1332us) server_negotiation.cc:407] Connection header received
0806 03:04:37.325033 (+ 1853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:37.325039 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:37.325119 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:37.325241 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:04:37.326436 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:37.327547 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:37.329889 (+ 2342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:37.330448 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:37.330650 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:37.332493 (+ 1843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:04:37.332503 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:04:37.332509 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:04:37.332536 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:04:37.332588 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:04:37.332608 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:04:37.332719 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:37.332996 (+ 277us) server_negotiation.cc:299] Negotiation successful
0806 03:04:37.333113 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":421,"thread_start_us":268,"threads_started":1}
I20260806 03:04:37.334774 18224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:37.321939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:37897 (local address 127.31.74.194:52943)
0806 03:04:37.323016 (+ 1077us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:37.323055 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 03:04:37.324762 (+ 1707us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:04:37.325452 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:04:37.325460 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:04:37.325554 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:04:37.326232 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:37.326250 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:37.327769 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:37.327775 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:37.329691 (+ 1916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:37.329706 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:37.330809 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:37.330819 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:37.331170 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:37.332030 (+ 860us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 03:04:37.332033 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:04:37.332106 (+ 73us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 03:04:37.332231 (+ 125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:04:37.332742 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:04:37.332752 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:04:37.332759 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 03:04:37.332953 (+ 194us) client_negotiation.cc:240] Negotiation successful
0806 03:04:37.333315 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":318,"threads_started":1}
I20260806 03:04:37.336608 18221 heartbeater.cc:347] Connected to a master server at 127.31.74.254:37897
I20260806 03:04:37.336953 18221 heartbeater.cc:464] Registering TS with master...
I20260806 03:04:37.338462 18221 heartbeater.cc:511] Master 127.31.74.254:37897 requested a full tablet report, sending...
I20260806 03:04:37.340498 17894 ts_manager.cc:194] Registered new tserver with Master: 0fbdeac3ff794b62abf26137d6d1076b (127.31.74.194:35477)
I20260806 03:04:37.341547 17894 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.74.194:52943
I20260806 03:04:37.741560 18084 heartbeater.cc:503] Master 127.31.74.254:37897 was elected leader, sending a full tablet report...
W20260806 03:04:37.749708 18225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:04:37.750092 18225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:04:37.750274 18225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:04:37.759532 18225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:04:37.759727 18225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:04:37.759923 18225 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.195
I20260806 03:04:37.770720 18225 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:44409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785985105255475-32043-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785985105255475-32043-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.74.195
--webserver_port=0
--tserver_master_addrs=127.31.74.254:37897
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.195
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785985105255475-32043-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:04:37.772166 18225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:04:37.774013 18225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:04:37.789816 18234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:04:37.790125 18232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:04:37.790172 18225 server_base.cc:1034] running on GCE node
W20260806 03:04:37.789852 18231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:04:37.791316 18225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:04:37.792461 18225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:04:37.793700 18225 hybrid_clock.cc:648] HybridClock initialized: now 1785985477793616 us; error 72 us; skew 500 ppm
I20260806 03:04:37.794083 18225 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 03:04:37.797199 18225 webserver.cc:460] Webserver started at http://127.31.74.195:40839/ using document root <none> and password file <none>
I20260806 03:04:37.797926 18225 fs_manager.cc:359] Metadata directory not provided
I20260806 03:04:37.798019 18225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:04:37.798296 18225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:04:37.802294 18225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785985105255475-32043-0/minicluster-data/ts-2/data/instance:
uuid: "8953071119af4d4cbd3f64a9801e5946"
format_stamp: "Formatted at 2026-08-06 03:04:37 on dist-test-slave-dhph"
server_key: "cb7fab70d96561c25088e23888120d07"
server_key_iv: "d516fbd034a13a08120421ecfa8f5c88"
server_key_version: "encryptionkey@0"
I20260806 03:04:37.803529 18225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785985105255475-32043-0/minicluster-data/ts-2/wal/instance:
uuid: "8953071119af4d4cbd3f64a9801e5946"
format_stamp: "Formatted at 2026-08-06 03:04:37 on dist-test-slave-dhph"
server_key: "cb7fab70d96561c25088e23888120d07"
server_key_iv: "d516fbd034a13a08120421ecfa8f5c88"
server_key_version: "encryptionkey@0"
I20260806 03:04:37.809649 18225 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260806 03:04:37.813831 18240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:04:37.815613 18225 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 03:04:37.815848 18225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785985105255475-32043-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785985105255475-32043-0/minicluster-data/ts-2/wal
uuid: "8953071119af4d4cbd3f64a9801e5946"
format_stamp: "Formatted at 2026-08-06 03:04:37 on dist-test-slave-dhph"
server_key: "cb7fab70d96561c25088e23888120d07"
server_key_iv: "d516fbd034a13a08120421ecfa8f5c88"
server_key_version: "encryptionkey@0"
I20260806 03:04:37.816103 18225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785985105255475-32043-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785985105255475-32043-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785985105255475-32043-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:04:37.838420 18225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:04:37.840063 18225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:04:37.840662 18225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:04:37.841874 18225 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:04:37.843786 18225 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:04:37.843946 18225 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:04:37.844115 18225 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:04:37.844220 18225 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:04:37.881687 18225 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.195:42699
I20260806 03:04:37.881785 18355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.195:42699 every 8 connection(s)
I20260806 03:04:37.883330 18225 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
I20260806 03:04:37.884624 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 18225
I20260806 03:04:37.884850 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785985105255475-32043-0/minicluster-data/ts-2/wal/instance
I20260806 03:04:37.885350 32043 external_mini_cluster.cc:1428] Setting key e155815af34f4be87aa2c812a238272d
I20260806 03:04:37.899384 18358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:37.887149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.195:35503 (local address 127.31.74.254:37897)
0806 03:04:37.887711 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:37.887718 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:37.889289 (+ 1571us) server_negotiation.cc:407] Connection header received
0806 03:04:37.891438 (+ 2149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:37.891444 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:37.891536 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:37.891672 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:04:37.893269 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:37.894104 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:37.896168 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:37.896669 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:37.896823 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:37.898413 (+ 1590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:04:37.898418 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:04:37.898422 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:04:37.898444 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:04:37.898485 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:04:37.898498 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:04:37.898623 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:37.898888 (+ 265us) server_negotiation.cc:299] Negotiation successful
0806 03:04:37.899001 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":276,"threads_started":1}
I20260806 03:04:37.900748 18359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:37.888052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:37897 (local address 127.31.74.195:35503)
0806 03:04:37.889120 (+ 1068us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:37.889168 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 03:04:37.891157 (+ 1989us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:04:37.891919 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:04:37.892157 (+ 238us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:04:37.892281 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:04:37.893010 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:37.893029 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:37.894294 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:37.894308 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:37.895974 (+ 1666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:37.895989 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:37.896929 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:37.896935 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:37.897190 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:37.898016 (+ 826us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 03:04:37.898019 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:04:37.898062 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 03:04:37.898174 (+ 112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:04:37.898630 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:04:37.898643 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:04:37.898653 (+ 10us) client_negotiation.cc:769] Sending connection context
0806 03:04:37.898879 (+ 226us) client_negotiation.cc:240] Negotiation successful
0806 03:04:37.899330 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":383,"threads_started":1}
I20260806 03:04:37.902729 18356 heartbeater.cc:347] Connected to a master server at 127.31.74.254:37897
I20260806 03:04:37.903072 18356 heartbeater.cc:464] Registering TS with master...
I20260806 03:04:37.903920 18356 heartbeater.cc:511] Master 127.31.74.254:37897 requested a full tablet report, sending...
I20260806 03:04:37.906158 17893 ts_manager.cc:194] Registered new tserver with Master: 8953071119af4d4cbd3f64a9801e5946 (127.31.74.195:42699)
I20260806 03:04:37.907406 17893 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.74.195:35503
I20260806 03:04:37.916980 32043 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 03:04:37.932622 18358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:37.923444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57062 (local address 127.31.74.254:37897)
0806 03:04:37.923720 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:37.923745 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:37.924374 (+ 629us) server_negotiation.cc:407] Connection header received
0806 03:04:37.924882 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:37.924887 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:37.924978 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:37.925087 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:04:37.926097 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:37.927390 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:37.929630 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:37.930070 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:37.930242 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:37.931680 (+ 1438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:04:37.931686 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:04:37.931690 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:04:37.931715 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:04:37.931752 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:04:37.931765 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:04:37.931878 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:37.932079 (+ 201us) server_negotiation.cc:299] Negotiation successful
0806 03:04:37.932194 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260806 03:04:37.937733 17893 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57062:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 03:04:37.947125 17893 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 03:04:37.999202 18370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:37.964209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.195:42699 (local address 127.0.0.1:56098)
0806 03:04:37.976045 (+ 11836us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:37.976111 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0806 03:04:37.976615 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:04:37.989113 (+ 12498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:04:37.989122 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:04:37.989158 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:04:37.989614 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:37.989627 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:37.992654 (+ 3027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:37.992659 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:37.995259 (+ 2600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:37.995272 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:37.997317 (+ 2045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:37.997324 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:37.998104 (+ 780us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:37.998194 (+ 90us) client_negotiation.cc:769] Sending connection context
0806 03:04:37.998364 (+ 170us) client_negotiation.cc:240] Negotiation successful
0806 03:04:37.998640 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11426,"mutex_wait_us":470,"thread_start_us":11189,"threads_started":1}
I20260806 03:04:37.999891 18373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:37.965015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56098 (local address 127.31.74.195:42699)
0806 03:04:37.988206 (+ 23191us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:37.988214 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:37.988252 (+ 38us) server_negotiation.cc:407] Connection header received
0806 03:04:37.988360 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:37.988368 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:37.988643 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:37.989842 (+ 1199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:04:37.990097 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:37.991771 (+ 1674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:37.995512 (+ 3741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:37.997083 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:37.997471 (+ 388us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:37.997784 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:37.999302 (+ 1518us) server_negotiation.cc:299] Negotiation successful
0806 03:04:37.999498 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23012,"thread_start_us":23788,"threads_started":1}
I20260806 03:04:38.003297 18371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:37.969118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:35439 (local address 127.0.0.1:41790)
0806 03:04:37.992358 (+ 23240us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:37.992380 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 03:04:37.992507 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:04:37.994121 (+ 1614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:04:37.994126 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:04:37.994157 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:04:37.994748 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:37.994764 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:37.997723 (+ 2959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:37.997730 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:38.000454 (+ 2724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:38.000472 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:38.002353 (+ 1881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:38.002362 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:38.002546 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:38.002637 (+ 91us) client_negotiation.cc:769] Sending connection context
0806 03:04:38.002765 (+ 128us) client_negotiation.cc:240] Negotiation successful
0806 03:04:38.002925 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23008,"thread_start_us":18717,"threads_started":1}
I20260806 03:04:38.003383 18374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:37.969314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41790 (local address 127.31.74.193:35439)
0806 03:04:37.987168 (+ 17854us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:37.987179 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:37.992622 (+ 5443us) server_negotiation.cc:407] Connection header received
0806 03:04:37.992944 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:37.992956 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:37.993443 (+ 487us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:37.993815 (+ 372us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:04:37.995022 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:37.997196 (+ 2174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:38.000747 (+ 3551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:38.002128 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:38.002308 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:38.002646 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:38.002838 (+ 192us) server_negotiation.cc:299] Negotiation successful
0806 03:04:38.003009 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17633,"thread_start_us":17853,"threads_started":1}
I20260806 03:04:38.005153 18372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:37.964895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40994 (local address 127.31.74.194:35477)
0806 03:04:37.979142 (+ 14247us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:37.979154 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:37.988538 (+ 9384us) server_negotiation.cc:407] Connection header received
0806 03:04:37.988999 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:37.989012 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:37.989307 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:37.989537 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:04:37.993863 (+ 4326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:37.995602 (+ 1739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:38.001518 (+ 5916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:38.003271 (+ 1753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:38.003435 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:38.003678 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:38.004302 (+ 624us) server_negotiation.cc:299] Negotiation successful
0806 03:04:38.004805 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14043,"thread_start_us":14270,"threads_started":1}
I20260806 03:04:38.005529 18375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:37.964459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.194:35477 (local address 127.0.0.1:40994)
0806 03:04:37.988330 (+ 23871us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:37.988362 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0806 03:04:37.988732 (+ 370us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:04:37.989992 (+ 1260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:04:37.990001 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:04:37.990042 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:04:37.993582 (+ 3540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:37.993601 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:37.995832 (+ 2231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:37.995838 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:38.001200 (+ 5362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:38.001216 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:38.003619 (+ 2403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:38.003626 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:38.003823 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:38.003862 (+ 39us) client_negotiation.cc:769] Sending connection context
0806 03:04:38.003945 (+ 83us) client_negotiation.cc:240] Negotiation successful
0806 03:04:38.004089 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23686,"thread_start_us":27577,"threads_started":1}
I20260806 03:04:38.005064 18289 tablet_service.cc:1467] Processing CreateTablet for tablet 153f5ec9aaf24dc6befc777ffb85dc02 (DEFAULT_TABLE table=test-table [id=ab010915c2c944bcb41d23463b2946f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:04:38.007505 18289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 153f5ec9aaf24dc6befc777ffb85dc02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:04:38.008220 18017 tablet_service.cc:1467] Processing CreateTablet for tablet 153f5ec9aaf24dc6befc777ffb85dc02 (DEFAULT_TABLE table=test-table [id=ab010915c2c944bcb41d23463b2946f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:04:38.009789 18017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 153f5ec9aaf24dc6befc777ffb85dc02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:04:38.009940 18154 tablet_service.cc:1467] Processing CreateTablet for tablet 153f5ec9aaf24dc6befc777ffb85dc02 (DEFAULT_TABLE table=test-table [id=ab010915c2c944bcb41d23463b2946f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:04:38.011353 18154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 153f5ec9aaf24dc6befc777ffb85dc02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:04:38.024787 18376 tablet_bootstrap.cc:492] T 153f5ec9aaf24dc6befc777ffb85dc02 P 55edf67c2f004e809fc591077349866b: Bootstrap starting.
I20260806 03:04:38.028546 18376 tablet_bootstrap.cc:654] T 153f5ec9aaf24dc6befc777ffb85dc02 P 55edf67c2f004e809fc591077349866b: Neither blocks nor log segments found. Creating new log.
I20260806 03:04:38.029994 18376 log.cc:824] T 153f5ec9aaf24dc6befc777ffb85dc02 P 55edf67c2f004e809fc591077349866b: Log is configured to *not* fsync() on all Append() calls
I20260806 03:04:38.031837 18377 tablet_bootstrap.cc:492] T 153f5ec9aaf24dc6befc777ffb85dc02 P 8953071119af4d4cbd3f64a9801e5946: Bootstrap starting.
I20260806 03:04:38.034610 18378 tablet_bootstrap.cc:492] T 153f5ec9aaf24dc6befc777ffb85dc02 P 0fbdeac3ff794b62abf26137d6d1076b: Bootstrap starting.
I20260806 03:04:38.036538 18377 tablet_bootstrap.cc:654] T 153f5ec9aaf24dc6befc777ffb85dc02 P 8953071119af4d4cbd3f64a9801e5946: Neither blocks nor log segments found. Creating new log.
I20260806 03:04:38.038185 18377 log.cc:824] T 153f5ec9aaf24dc6befc777ffb85dc02 P 8953071119af4d4cbd3f64a9801e5946: Log is configured to *not* fsync() on all Append() calls
I20260806 03:04:38.039407 18378 tablet_bootstrap.cc:654] T 153f5ec9aaf24dc6befc777ffb85dc02 P 0fbdeac3ff794b62abf26137d6d1076b: Neither blocks nor log segments found. Creating new log.
I20260806 03:04:38.039749 18376 tablet_bootstrap.cc:492] T 153f5ec9aaf24dc6befc777ffb85dc02 P 55edf67c2f004e809fc591077349866b: No bootstrap required, opened a new log
I20260806 03:04:38.040102 18376 ts_tablet_manager.cc:1397] T 153f5ec9aaf24dc6befc777ffb85dc02 P 55edf67c2f004e809fc591077349866b: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.006s
I20260806 03:04:38.041440 18378 log.cc:824] T 153f5ec9aaf24dc6befc777ffb85dc02 P 0fbdeac3ff794b62abf26137d6d1076b: Log is configured to *not* fsync() on all Append() calls
I20260806 03:04:38.043042 18377 tablet_bootstrap.cc:492] T 153f5ec9aaf24dc6befc777ffb85dc02 P 8953071119af4d4cbd3f64a9801e5946: No bootstrap required, opened a new log
I20260806 03:04:38.043388 18377 ts_tablet_manager.cc:1397] T 153f5ec9aaf24dc6befc777ffb85dc02 P 8953071119af4d4cbd3f64a9801e5946: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260806 03:04:38.045044 18378 tablet_bootstrap.cc:492] T 153f5ec9aaf24dc6befc777ffb85dc02 P 0fbdeac3ff794b62abf26137d6d1076b: No bootstrap required, opened a new log
I20260806 03:04:38.045344 18378 ts_tablet_manager.cc:1397] T 153f5ec9aaf24dc6befc777ffb85dc02 P 0fbdeac3ff794b62abf26137d6d1076b: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260806 03:04:38.049229 18376 raft_consensus.cc:348] T 153f5ec9aaf24dc6befc777ffb85dc02 P 55edf67c2f004e809fc591077349866b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8953071119af4d4cbd3f64a9801e5946" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 42699 } } peers { permanent_uuid: "0fbdeac3ff794b62abf26137d6d1076b" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 35477 } } peers { permanent_uuid: "55edf67c2f004e809fc591077349866b" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 35439 } }
I20260806 03:04:38.049660 18376 raft_consensus.cc:374] T 153f5ec9aaf24dc6befc777ffb85dc02 P 55edf67c2f004e809fc591077349866b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:04:38.049770 18376 raft_consensus.cc:729] T 153f5ec9aaf24dc6befc777ffb85dc02 P 55edf67c2f004e809fc591077349866b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55edf67c2f004e809fc591077349866b, State: Initialized, Role: FOLLOWER
I20260806 03:04:38.050457 18376 consensus_queue.cc:260] T 153f5ec9aaf24dc6befc777ffb85dc02 P 55edf67c2f004e809fc591077349866b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8953071119af4d4cbd3f64a9801e5946" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 42699 } } peers { permanent_uuid: "0fbdeac3ff794b62abf26137d6d1076b" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 35477 } } peers { permanent_uuid: "55edf67c2f004e809fc591077349866b" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 35439 } }
I20260806 03:04:38.052297 18376 ts_tablet_manager.cc:1428] T 153f5ec9aaf24dc6befc777ffb85dc02 P 55edf67c2f004e809fc591077349866b: Time spent starting tablet: real 0.012s user 0.012s sys 0.001s
I20260806 03:04:38.055068 18377 raft_consensus.cc:348] T 153f5ec9aaf24dc6befc777ffb85dc02 P 8953071119af4d4cbd3f64a9801e5946 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8953071119af4d4cbd3f64a9801e5946" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 42699 } } peers { permanent_uuid: "0fbdeac3ff794b62abf26137d6d1076b" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 35477 } } peers { permanent_uuid: "55edf67c2f004e809fc591077349866b" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 35439 } }
I20260806 03:04:38.055478 18377 raft_consensus.cc:374] T 153f5ec9aaf24dc6befc777ffb85dc02 P 8953071119af4d4cbd3f64a9801e5946 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:04:38.055603 18377 raft_consensus.cc:729] T 153f5ec9aaf24dc6befc777ffb85dc02 P 8953071119af4d4cbd3f64a9801e5946 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8953071119af4d4cbd3f64a9801e5946, State: Initialized, Role: FOLLOWER
I20260806 03:04:38.056308 18377 consensus_queue.cc:260] T 153f5ec9aaf24dc6befc777ffb85dc02 P 8953071119af4d4cbd3f64a9801e5946 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8953071119af4d4cbd3f64a9801e5946" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 42699 } } peers { permanent_uuid: "0fbdeac3ff794b62abf26137d6d1076b" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 35477 } } peers { permanent_uuid: "55edf67c2f004e809fc591077349866b" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 35439 } }
I20260806 03:04:38.057255 18378 raft_consensus.cc:348] T 153f5ec9aaf24dc6befc777ffb85dc02 P 0fbdeac3ff794b62abf26137d6d1076b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8953071119af4d4cbd3f64a9801e5946" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 42699 } } peers { permanent_uuid: "0fbdeac3ff794b62abf26137d6d1076b" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 35477 } } peers { permanent_uuid: "55edf67c2f004e809fc591077349866b" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 35439 } }
I20260806 03:04:38.057771 18378 raft_consensus.cc:374] T 153f5ec9aaf24dc6befc777ffb85dc02 P 0fbdeac3ff794b62abf26137d6d1076b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:04:38.057878 18378 raft_consensus.cc:729] T 153f5ec9aaf24dc6befc777ffb85dc02 P 0fbdeac3ff794b62abf26137d6d1076b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fbdeac3ff794b62abf26137d6d1076b, State: Initialized, Role: FOLLOWER
I20260806 03:04:38.057770 18356 heartbeater.cc:503] Master 127.31.74.254:37897 was elected leader, sending a full tablet report...
I20260806 03:04:38.058513 18378 consensus_queue.cc:260] T 153f5ec9aaf24dc6befc777ffb85dc02 P 0fbdeac3ff794b62abf26137d6d1076b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8953071119af4d4cbd3f64a9801e5946" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 42699 } } peers { permanent_uuid: "0fbdeac3ff794b62abf26137d6d1076b" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 35477 } } peers { permanent_uuid: "55edf67c2f004e809fc591077349866b" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 35439 } }
I20260806 03:04:38.059738 18377 ts_tablet_manager.cc:1428] T 153f5ec9aaf24dc6befc777ffb85dc02 P 8953071119af4d4cbd3f64a9801e5946: Time spent starting tablet: real 0.016s user 0.010s sys 0.004s
I20260806 03:04:38.060910 18221 heartbeater.cc:503] Master 127.31.74.254:37897 was elected leader, sending a full tablet report...
I20260806 03:04:38.061632 18378 ts_tablet_manager.cc:1428] T 153f5ec9aaf24dc6befc777ffb85dc02 P 0fbdeac3ff794b62abf26137d6d1076b: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
W20260806 03:04:38.070492 18222 tablet.cc:2369] T 153f5ec9aaf24dc6befc777ffb85dc02 P 0fbdeac3ff794b62abf26137d6d1076b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 03:04:38.118116 18383 raft_consensus.cc:482] T 153f5ec9aaf24dc6befc777ffb85dc02 P 8953071119af4d4cbd3f64a9801e5946 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:04:38.118571 18383 raft_consensus.cc:504] T 153f5ec9aaf24dc6befc777ffb85dc02 P 8953071119af4d4cbd3f64a9801e5946 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8953071119af4d4cbd3f64a9801e5946" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 42699 } } peers { permanent_uuid: "0fbdeac3ff794b62abf26137d6d1076b" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 35477 } } peers { permanent_uuid: "55edf67c2f004e809fc591077349866b" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 35439 } }
I20260806 03:04:38.120909 18383 leader_election.cc:288] T 153f5ec9aaf24dc6befc777ffb85dc02 P 8953071119af4d4cbd3f64a9801e5946 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fbdeac3ff794b62abf26137d6d1076b (127.31.74.194:35477), 55edf67c2f004e809fc591077349866b (127.31.74.193:35439)
I20260806 03:04:38.129739 18374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:38.121850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.195:56537 (local address 127.31.74.193:35439)
0806 03:04:38.122051 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:38.122057 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:38.122078 (+ 21us) server_negotiation.cc:407] Connection header received
0806 03:04:38.122165 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:38.122170 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:38.122262 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:38.122341 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:04:38.123376 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:38.124146 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:38.127222 (+ 3076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:38.128523 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:38.128712 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:38.128789 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:38.129299 (+ 510us) server_negotiation.cc:299] Negotiation successful
0806 03:04:38.129463 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 03:04:38.129782 18359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:38.121199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:35439 (local address 127.31.74.195:56537)
0806 03:04:38.121438 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:38.121462 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 03:04:38.121645 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:04:38.122815 (+ 1170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:04:38.122822 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:04:38.122866 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:04:38.123179 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:38.123190 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:38.124387 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:38.124393 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:38.126759 (+ 2366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:38.126777 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:38.128697 (+ 1920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:38.128704 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:38.128968 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:38.129145 (+ 177us) client_negotiation.cc:769] Sending connection context
0806 03:04:38.129272 (+ 127us) client_negotiation.cc:240] Negotiation successful
0806 03:04:38.129482 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81}
I20260806 03:04:38.130090 18372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:38.121693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.195:42869 (local address 127.31.74.194:35477)
0806 03:04:38.121959 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:38.121965 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:38.122199 (+ 234us) server_negotiation.cc:407] Connection header received
0806 03:04:38.122421 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:38.122427 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:38.122514 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:38.122647 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:04:38.124379 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:38.125180 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:38.128026 (+ 2846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:38.128900 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:38.129082 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:38.129167 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:38.129630 (+ 463us) server_negotiation.cc:299] Negotiation successful
0806 03:04:38.129774 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260806 03:04:38.130640 18385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:38.121298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.194:35477 (local address 127.31.74.195:42869)
0806 03:04:38.122094 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:38.122113 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 03:04:38.122259 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:04:38.122674 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:04:38.122679 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:04:38.122707 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:04:38.123167 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:38.123179 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:38.125371 (+ 2192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:38.125406 (+ 35us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:38.127793 (+ 2387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:38.127807 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:38.129134 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:38.129141 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:38.129313 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:38.129340 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 03:04:38.129568 (+ 228us) client_negotiation.cc:240] Negotiation successful
0806 03:04:38.129732 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":666,"thread_start_us":626,"threads_started":1}
I20260806 03:04:38.131062 18038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "153f5ec9aaf24dc6befc777ffb85dc02" candidate_uuid: "8953071119af4d4cbd3f64a9801e5946" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55edf67c2f004e809fc591077349866b" is_pre_election: true
I20260806 03:04:38.131673 18038 raft_consensus.cc:2454] T 153f5ec9aaf24dc6befc777ffb85dc02 P 55edf67c2f004e809fc591077349866b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8953071119af4d4cbd3f64a9801e5946 in term 0.
I20260806 03:04:38.132035 18174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "153f5ec9aaf24dc6befc777ffb85dc02" candidate_uuid: "8953071119af4d4cbd3f64a9801e5946" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbdeac3ff794b62abf26137d6d1076b" is_pre_election: true
I20260806 03:04:38.132604 18174 raft_consensus.cc:2454] T 153f5ec9aaf24dc6befc777ffb85dc02 P 0fbdeac3ff794b62abf26137d6d1076b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8953071119af4d4cbd3f64a9801e5946 in term 0.
I20260806 03:04:38.132586 18242 leader_election.cc:302] T 153f5ec9aaf24dc6befc777ffb85dc02 P 8953071119af4d4cbd3f64a9801e5946 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55edf67c2f004e809fc591077349866b, 8953071119af4d4cbd3f64a9801e5946; no voters:
I20260806 03:04:38.133368 18383 raft_consensus.cc:2781] T 153f5ec9aaf24dc6befc777ffb85dc02 P 8953071119af4d4cbd3f64a9801e5946 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:04:38.133533 18383 raft_consensus.cc:482] T 153f5ec9aaf24dc6befc777ffb85dc02 P 8953071119af4d4cbd3f64a9801e5946 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:04:38.133644 18383 raft_consensus.cc:3037] T 153f5ec9aaf24dc6befc777ffb85dc02 P 8953071119af4d4cbd3f64a9801e5946 [term 0 FOLLOWER]: Advancing to term 1
W20260806 03:04:38.135550 18357 tablet.cc:2369] T 153f5ec9aaf24dc6befc777ffb85dc02 P 8953071119af4d4cbd3f64a9801e5946: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 03:04:38.136307 18383 raft_consensus.cc:504] T 153f5ec9aaf24dc6befc777ffb85dc02 P 8953071119af4d4cbd3f64a9801e5946 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8953071119af4d4cbd3f64a9801e5946" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 42699 } } peers { permanent_uuid: "0fbdeac3ff794b62abf26137d6d1076b" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 35477 } } peers { permanent_uuid: "55edf67c2f004e809fc591077349866b" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 35439 } }
I20260806 03:04:38.137028 18383 leader_election.cc:288] T 153f5ec9aaf24dc6befc777ffb85dc02 P 8953071119af4d4cbd3f64a9801e5946 [CANDIDATE]: Term 1 election: Requested vote from peers 0fbdeac3ff794b62abf26137d6d1076b (127.31.74.194:35477), 55edf67c2f004e809fc591077349866b (127.31.74.193:35439)
I20260806 03:04:38.137555 18174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "153f5ec9aaf24dc6befc777ffb85dc02" candidate_uuid: "8953071119af4d4cbd3f64a9801e5946" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fbdeac3ff794b62abf26137d6d1076b"
I20260806 03:04:38.137778 18174 raft_consensus.cc:3037] T 153f5ec9aaf24dc6befc777ffb85dc02 P 0fbdeac3ff794b62abf26137d6d1076b [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:04:38.137686 18038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "153f5ec9aaf24dc6befc777ffb85dc02" candidate_uuid: "8953071119af4d4cbd3f64a9801e5946" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55edf67c2f004e809fc591077349866b"
I20260806 03:04:38.138016 18038 raft_consensus.cc:3037] T 153f5ec9aaf24dc6befc777ffb85dc02 P 55edf67c2f004e809fc591077349866b [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:04:38.140551 18174 raft_consensus.cc:2454] T 153f5ec9aaf24dc6befc777ffb85dc02 P 0fbdeac3ff794b62abf26137d6d1076b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8953071119af4d4cbd3f64a9801e5946 in term 1.
I20260806 03:04:38.141031 18243 leader_election.cc:302] T 153f5ec9aaf24dc6befc777ffb85dc02 P 8953071119af4d4cbd3f64a9801e5946 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fbdeac3ff794b62abf26137d6d1076b, 8953071119af4d4cbd3f64a9801e5946; no voters:
I20260806 03:04:38.141474 18383 raft_consensus.cc:2781] T 153f5ec9aaf24dc6befc777ffb85dc02 P 8953071119af4d4cbd3f64a9801e5946 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:04:38.141616 18038 raft_consensus.cc:2454] T 153f5ec9aaf24dc6befc777ffb85dc02 P 55edf67c2f004e809fc591077349866b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8953071119af4d4cbd3f64a9801e5946 in term 1.
I20260806 03:04:38.142120 18383 raft_consensus.cc:686] T 153f5ec9aaf24dc6befc777ffb85dc02 P 8953071119af4d4cbd3f64a9801e5946 [term 1 LEADER]: Becoming Leader. State: Replica: 8953071119af4d4cbd3f64a9801e5946, State: Running, Role: LEADER
I20260806 03:04:38.142736 18383 consensus_queue.cc:237] T 153f5ec9aaf24dc6befc777ffb85dc02 P 8953071119af4d4cbd3f64a9801e5946 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8953071119af4d4cbd3f64a9801e5946" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 42699 } } peers { permanent_uuid: "0fbdeac3ff794b62abf26137d6d1076b" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 35477 } } peers { permanent_uuid: "55edf67c2f004e809fc591077349866b" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 35439 } }
I20260806 03:04:38.148851 17894 catalog_manager.cc:5495] T 153f5ec9aaf24dc6befc777ffb85dc02 P 8953071119af4d4cbd3f64a9801e5946 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8953071119af4d4cbd3f64a9801e5946 (127.31.74.195). New cstate: current_term: 1 leader_uuid: "8953071119af4d4cbd3f64a9801e5946" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8953071119af4d4cbd3f64a9801e5946" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 42699 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0fbdeac3ff794b62abf26137d6d1076b" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 35477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "55edf67c2f004e809fc591077349866b" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 35439 } health_report { overall_health: UNKNOWN } } }
I20260806 03:04:38.192453 18373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:38.185291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56100 (local address 127.31.74.195:42699)
0806 03:04:38.185553 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:38.185560 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:38.185690 (+ 130us) server_negotiation.cc:407] Connection header received
0806 03:04:38.185978 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:38.185982 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:38.186181 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:38.186551 (+ 370us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 03:04:38.187049 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:38.187841 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:38.189803 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:38.190328 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:38.190480 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:38.191208 (+ 728us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 03:04:38.191802 (+ 594us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 03:04:38.191964 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:38.192051 (+ 87us) server_negotiation.cc:299] Negotiation successful
0806 03:04:38.192142 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260806 03:04:38.203619 18038 raft_consensus.cc:1261] T 153f5ec9aaf24dc6befc777ffb85dc02 P 55edf67c2f004e809fc591077349866b [term 1 FOLLOWER]: Refusing update from remote peer 8953071119af4d4cbd3f64a9801e5946: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:04:38.203619 18174 raft_consensus.cc:1261] T 153f5ec9aaf24dc6befc777ffb85dc02 P 0fbdeac3ff794b62abf26137d6d1076b [term 1 FOLLOWER]: Refusing update from remote peer 8953071119af4d4cbd3f64a9801e5946: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:04:38.204754 18386 consensus_queue.cc:1035] T 153f5ec9aaf24dc6befc777ffb85dc02 P 8953071119af4d4cbd3f64a9801e5946 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55edf67c2f004e809fc591077349866b" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 35439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:04:38.205178 18383 consensus_queue.cc:1035] T 153f5ec9aaf24dc6befc777ffb85dc02 P 8953071119af4d4cbd3f64a9801e5946 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fbdeac3ff794b62abf26137d6d1076b" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 35477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 03:04:38.207526 18086 tablet.cc:2369] T 153f5ec9aaf24dc6befc777ffb85dc02 P 55edf67c2f004e809fc591077349866b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 03:04:38.221575 18389 mvcc.cc:205] Tried to move back new op lower bound from 7315396518707068928 to 7315396518476681216. Current Snapshot: MvccSnapshot[applied={T|T < 7315396518707068928}]
I20260806 03:04:38.227169 18391 mvcc.cc:205] Tried to move back new op lower bound from 7315396518707068928 to 7315396518476681216. Current Snapshot: MvccSnapshot[applied={T|T < 7315396518707068928}]
I20260806 03:04:38.230250 18390 mvcc.cc:205] Tried to move back new op lower bound from 7315396518707068928 to 7315396518476681216. Current Snapshot: MvccSnapshot[applied={T|T < 7315396518707068928}]
I20260806 03:04:38.245086 17893 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57062:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 03:04:38.245438 17893 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57062:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 03:04:38.249975 17893 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e976b933af1443328d21321228453ed1: Sending DeleteTablet for 3 replicas of tablet 153f5ec9aaf24dc6befc777ffb85dc02
I20260806 03:04:38.251319 18289 tablet_service.cc:1514] Processing DeleteTablet for tablet 153f5ec9aaf24dc6befc777ffb85dc02 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:04:38 UTC) from {username='slave'} at 127.0.0.1:56098
I20260806 03:04:38.251367 18017 tablet_service.cc:1514] Processing DeleteTablet for tablet 153f5ec9aaf24dc6befc777ffb85dc02 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:04:38 UTC) from {username='slave'} at 127.0.0.1:41790
I20260806 03:04:38.251360 18154 tablet_service.cc:1514] Processing DeleteTablet for tablet 153f5ec9aaf24dc6befc777ffb85dc02 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:04:38 UTC) from {username='slave'} at 127.0.0.1:40994
I20260806 03:04:38.255321 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 17955
I20260806 03:04:38.255385 18399 tablet_replica.cc:330] T 153f5ec9aaf24dc6befc777ffb85dc02 P 55edf67c2f004e809fc591077349866b: stopping tablet replica
I20260806 03:04:38.256024 18399 raft_consensus.cc:2229] T 153f5ec9aaf24dc6befc777ffb85dc02 P 55edf67c2f004e809fc591077349866b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:04:38.256779 18399 raft_consensus.cc:2258] T 153f5ec9aaf24dc6befc777ffb85dc02 P 55edf67c2f004e809fc591077349866b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:04:38.261935 18399 ts_tablet_manager.cc:1905] T 153f5ec9aaf24dc6befc777ffb85dc02 P 55edf67c2f004e809fc591077349866b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:04:38.263773 18400 tablet_replica.cc:330] T 153f5ec9aaf24dc6befc777ffb85dc02 P 8953071119af4d4cbd3f64a9801e5946: stopping tablet replica
I20260806 03:04:38.264330 18400 raft_consensus.cc:2229] T 153f5ec9aaf24dc6befc777ffb85dc02 P 8953071119af4d4cbd3f64a9801e5946 [term 1 LEADER]: Raft consensus shutting down.
I20260806 03:04:38.266058 18400 raft_consensus.cc:2258] T 153f5ec9aaf24dc6befc777ffb85dc02 P 8953071119af4d4cbd3f64a9801e5946 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:04:38.267690 18401 tablet_replica.cc:330] T 153f5ec9aaf24dc6befc777ffb85dc02 P 0fbdeac3ff794b62abf26137d6d1076b: stopping tablet replica
I20260806 03:04:38.268339 18401 raft_consensus.cc:2229] T 153f5ec9aaf24dc6befc777ffb85dc02 P 0fbdeac3ff794b62abf26137d6d1076b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:04:38.268971 18401 raft_consensus.cc:2258] T 153f5ec9aaf24dc6befc777ffb85dc02 P 0fbdeac3ff794b62abf26137d6d1076b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:04:38.271198 18401 ts_tablet_manager.cc:1905] T 153f5ec9aaf24dc6befc777ffb85dc02 P 0fbdeac3ff794b62abf26137d6d1076b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:04:38.271216 18400 ts_tablet_manager.cc:1905] T 153f5ec9aaf24dc6befc777ffb85dc02 P 8953071119af4d4cbd3f64a9801e5946: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:04:38.275534 18374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:38.256904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41802 (local address 127.31.74.193:35439)
0806 03:04:38.257087 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:38.257093 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:38.264028 (+ 6935us) server_negotiation.cc:407] Connection header received
0806 03:04:38.264393 (+ 365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:38.264399 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:38.264519 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:38.264610 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:04:38.266276 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:38.268283 (+ 2007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:38.270748 (+ 2465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:38.271420 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:38.271536 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:38.273380 (+ 1844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:04:38.273396 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:04:38.273414 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:04:38.274267 (+ 853us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:04:38.274340 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:04:38.274371 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:04:38.274472 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:38.274822 (+ 350us) server_negotiation.cc:299] Negotiation successful
0806 03:04:38.274977 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 03:04:38.277132 18079 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 03:04:38.279243 18401 ts_tablet_manager.cc:1918] T 153f5ec9aaf24dc6befc777ffb85dc02 P 0fbdeac3ff794b62abf26137d6d1076b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 03:04:38.279527 18401 log.cc:1196] T 153f5ec9aaf24dc6befc777ffb85dc02 P 0fbdeac3ff794b62abf26137d6d1076b: Deleting WAL directory at /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785985105255475-32043-0/minicluster-data/ts-1/wal/wals/153f5ec9aaf24dc6befc777ffb85dc02
I20260806 03:04:38.280572 18401 ts_tablet_manager.cc:1939] T 153f5ec9aaf24dc6befc777ffb85dc02 P 0fbdeac3ff794b62abf26137d6d1076b: Deleting consensus metadata
I20260806 03:04:38.282902 18400 ts_tablet_manager.cc:1918] T 153f5ec9aaf24dc6befc777ffb85dc02 P 8953071119af4d4cbd3f64a9801e5946: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 03:04:38.283123 18400 log.cc:1196] T 153f5ec9aaf24dc6befc777ffb85dc02 P 8953071119af4d4cbd3f64a9801e5946: Deleting WAL directory at /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785985105255475-32043-0/minicluster-data/ts-2/wal/wals/153f5ec9aaf24dc6befc777ffb85dc02
I20260806 03:04:38.283179 17879 catalog_manager.cc:4841] TS 0fbdeac3ff794b62abf26137d6d1076b (127.31.74.194:35477): tablet 153f5ec9aaf24dc6befc777ffb85dc02 (table test-table [id=ab010915c2c944bcb41d23463b2946f8]) successfully deleted
I20260806 03:04:38.283705 18400 ts_tablet_manager.cc:1939] T 153f5ec9aaf24dc6befc777ffb85dc02 P 8953071119af4d4cbd3f64a9801e5946: Deleting consensus metadata
I20260806 03:04:38.285426 17877 catalog_manager.cc:4841] TS 8953071119af4d4cbd3f64a9801e5946 (127.31.74.195:42699): tablet 153f5ec9aaf24dc6befc777ffb85dc02 (table test-table [id=ab010915c2c944bcb41d23463b2946f8]) successfully deleted
I20260806 03:04:38.389708 18399 ts_tablet_manager.cc:1918] T 153f5ec9aaf24dc6befc777ffb85dc02 P 55edf67c2f004e809fc591077349866b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 03:04:38.390038 18399 log.cc:1196] T 153f5ec9aaf24dc6befc777ffb85dc02 P 55edf67c2f004e809fc591077349866b: Deleting WAL directory at /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785985105255475-32043-0/minicluster-data/ts-0/wal/wals/153f5ec9aaf24dc6befc777ffb85dc02
I20260806 03:04:38.390676 18399 ts_tablet_manager.cc:1939] T 153f5ec9aaf24dc6befc777ffb85dc02 P 55edf67c2f004e809fc591077349866b: Deleting consensus metadata
I20260806 03:04:38.390847 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 17955
I20260806 03:04:38.414160 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 18089
W20260806 03:04:38.414129 17878 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.74.193:35439 (error 108)
I20260806 03:04:38.416424 18375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:38.415915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:35439 (local address 127.0.0.1:41812)
0806 03:04:38.416110 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:38.416220 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.193:35439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260806 03:04:38.416843 17878 catalog_manager.cc:4584] TS 55edf67c2f004e809fc591077349866b (127.31.74.193:35439): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 153f5ec9aaf24dc6befc777ffb85dc02: Network error: Client connection negotiation failed: client connection to 127.31.74.193:35439: connect: Connection refused (error 111)
I20260806 03:04:38.425680 18372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:38.415381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40996 (local address 127.31.74.194:35477)
0806 03:04:38.415581 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:38.415587 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:38.415654 (+ 67us) server_negotiation.cc:407] Connection header received
0806 03:04:38.416338 (+ 684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:38.416343 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:38.416472 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:38.416584 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:04:38.417536 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:38.418508 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:38.420545 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:38.421163 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:38.421315 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:38.423630 (+ 2315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:04:38.423741 (+ 111us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:04:38.423762 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:04:38.424465 (+ 703us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:04:38.424592 (+ 127us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:04:38.424632 (+ 40us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:04:38.424786 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:38.424947 (+ 161us) server_negotiation.cc:299] Negotiation successful
0806 03:04:38.425168 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 03:04:38.426539 18216 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 03:04:38.470953 18375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:38.470188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:35439 (local address 127.0.0.1:41818)
0806 03:04:38.470573 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:38.470721 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.193:35439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219}
I20260806 03:04:38.472278 18375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:38.471814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:35439 (local address 127.0.0.1:41826)
0806 03:04:38.472006 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:38.472113 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.193:35439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260806 03:04:38.532335 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 18089
I20260806 03:04:38.533771 18375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:38.532964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:35439 (local address 127.0.0.1:41828)
0806 03:04:38.533313 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:38.533455 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.193:35439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194}
I20260806 03:04:38.535249 18375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:38.534714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:35439 (local address 127.0.0.1:41834)
0806 03:04:38.534978 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:38.535078 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.193:35439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148}
I20260806 03:04:38.558636 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 18225
I20260806 03:04:38.568578 18373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:38.560473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56106 (local address 127.31.74.195:42699)
0806 03:04:38.560724 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:38.560729 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:38.560752 (+ 23us) server_negotiation.cc:407] Connection header received
0806 03:04:38.560813 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:38.560816 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:38.560895 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:38.561020 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:04:38.561541 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:38.562341 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:38.563993 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:38.564773 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:38.565359 (+ 586us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:38.566909 (+ 1550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:04:38.566919 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:04:38.566939 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:04:38.567534 (+ 595us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:04:38.567592 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:04:38.567614 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:04:38.567747 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:38.567910 (+ 163us) server_negotiation.cc:299] Negotiation successful
0806 03:04:38.568026 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260806 03:04:38.569979 18351 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 03:04:38.616696 18375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:38.615817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:35439 (local address 127.0.0.1:41842)
0806 03:04:38.616148 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:38.616272 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.193:35439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181}
I20260806 03:04:38.618063 18375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:38.617644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:35439 (local address 127.0.0.1:41850)
0806 03:04:38.617864 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:38.617942 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.193:35439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260806 03:04:38.709518 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 18225
I20260806 03:04:38.735234 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 17862
I20260806 03:04:38.736629 17925 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 03:04:38.867020 18375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:38.866111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:35439 (local address 127.0.0.1:41862)
0806 03:04:38.866617 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:38.866789 (+ 172us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.193:35439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260806 03:04:38.867362 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 17862
2026-08-06T03:04:38Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3627 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 03:04:39 dist-test-slave-dhph krb5kdc[18410](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 03:04:39 dist-test-slave-dhph krb5kdc[18410](info): set up 2 sockets
Aug 06 03:04:39 dist-test-slave-dhph krb5kdc[18410](info): commencing operation
krb5kdc: starting...
W20260806 03:04:40.187281 32043 mini_kdc.cc:121] Time spent starting KDC: real 1.267s user 0.002s sys 0.058s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/krb5kdc/test-user.keytab.
Aug 06 03:04:40 dist-test-slave-dhph krb5kdc[18410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985480, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T03:04:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:04:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.74.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab.
Entry for principal kudu/127.31.74.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab.
I20260806 03:04:40.590663 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:40781
--webserver_interface=127.31.74.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab
--principal=kudu/127.31.74.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:39993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.74.254:40781
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:04:40.998608 18424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:04:40.999022 18424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:04:40.999130 18424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:04:41.007788 18424 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:04:41.007948 18424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:04:41.007998 18424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:04:41.008052 18424 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:04:41.008090 18424 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:04:41.018823 18424 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:39993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.74.254:40781
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab
--principal=kudu/127.31.74.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:40781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.74.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:04:41.020206 18424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:04:41.021587 18424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:04:41.035153 18430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:04:41.035216 18431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:04:41.035239 18433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:04:41.035812 18424 server_base.cc:1034] running on GCE node
I20260806 03:04:41.036944 18424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:04:41.039017 18424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:04:41.040351 18424 hybrid_clock.cc:648] HybridClock initialized: now 1785985481040298 us; error 106 us; skew 500 ppm
I20260806 03:04:41.040796 18424 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 03:04:41 dist-test-slave-dhph krb5kdc[18410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985481, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:04:41.044365 18424 init.cc:377] Logged in from keytab as kudu/127.31.74.254@KRBTEST.COM (short username kudu)
I20260806 03:04:41.046155 18424 webserver.cc:460] Webserver started at http://127.31.74.254:41463/ using document root <none> and password file <none>
I20260806 03:04:41.047029 18424 fs_manager.cc:359] Metadata directory not provided
I20260806 03:04:41.047233 18424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:04:41.047670 18424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:04:41.050992 18424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/minicluster-data/master-0/data/instance:
uuid: "32ac7f36a4444784a5e84c7f6304c127"
format_stamp: "Formatted at 2026-08-06 03:04:41 on dist-test-slave-dhph"
server_key: "6104c472ca8608521764a57706134a38"
server_key_iv: "f3618a20b750453055357eb3500f48d1"
server_key_version: "encryptionkey@0"
I20260806 03:04:41.051791 18424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/minicluster-data/master-0/wal/instance:
uuid: "32ac7f36a4444784a5e84c7f6304c127"
format_stamp: "Formatted at 2026-08-06 03:04:41 on dist-test-slave-dhph"
server_key: "6104c472ca8608521764a57706134a38"
server_key_iv: "f3618a20b750453055357eb3500f48d1"
server_key_version: "encryptionkey@0"
I20260806 03:04:41.057611 18424 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260806 03:04:41.062459 18440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:04:41.064312 18424 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 03:04:41.064544 18424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/minicluster-data/master-0/wal
uuid: "32ac7f36a4444784a5e84c7f6304c127"
format_stamp: "Formatted at 2026-08-06 03:04:41 on dist-test-slave-dhph"
server_key: "6104c472ca8608521764a57706134a38"
server_key_iv: "f3618a20b750453055357eb3500f48d1"
server_key_version: "encryptionkey@0"
I20260806 03:04:41.064791 18424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:04:41.095160 18424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:04:41.100878 18424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:04:41.101234 18424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:04:41.125829 18424 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.254:40781
I20260806 03:04:41.125831 18494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.254:40781 every 8 connection(s)
I20260806 03:04:41.127656 18424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
I20260806 03:04:41.132189 18495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:04:41.133388 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 18424
I20260806 03:04:41.133581 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/minicluster-data/master-0/wal/instance
I20260806 03:04:41.134186 32043 external_mini_cluster.cc:1428] Setting key 4b2eee58e0ac22783d4e8f5d2c396012
I20260806 03:04:41.148594 18495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32ac7f36a4444784a5e84c7f6304c127: Bootstrap starting.
Aug 06 03:04:41 dist-test-slave-dhph krb5kdc[18410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985480, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 03:04:41.154026 18495 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32ac7f36a4444784a5e84c7f6304c127: Neither blocks nor log segments found. Creating new log.
I20260806 03:04:41.155380 18495 log.cc:824] T 00000000000000000000000000000000 P 32ac7f36a4444784a5e84c7f6304c127: Log is configured to *not* fsync() on all Append() calls
I20260806 03:04:41.159685 18495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32ac7f36a4444784a5e84c7f6304c127: No bootstrap required, opened a new log
I20260806 03:04:41.166388 18498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:41.136329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45136 (local address 127.31.74.254:40781)
0806 03:04:41.137192 (+ 863us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:41.137203 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:41.137250 (+ 47us) server_negotiation.cc:407] Connection header received
0806 03:04:41.139042 (+ 1792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:41.139065 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:41.139476 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:41.140044 (+ 568us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:04:41.142299 (+ 2255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:41.144303 (+ 2004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:41.146865 (+ 2562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:41.147683 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:41.147859 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:41.154529 (+ 6670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:04:41.154548 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:04:41.154570 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:04:41.154687 (+ 117us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:04:41.159040 (+ 4353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:04:41.160600 (+ 1560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:04:41.160686 (+ 86us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:04:41.160694 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:04:41.160902 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:04:41.162099 (+ 1197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:04:41.162105 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:04:41.162108 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:04:41.163313 (+ 1205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:04:41.163829 (+ 516us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:41.164431 (+ 602us) server_negotiation.cc:299] Negotiation successful
0806 03:04:41.165089 (+ 658us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":403,"threads_started":1}
I20260806 03:04:41.171344 18495 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32ac7f36a4444784a5e84c7f6304c127 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32ac7f36a4444784a5e84c7f6304c127" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 40781 } }
I20260806 03:04:41.171743 18495 raft_consensus.cc:374] T 00000000000000000000000000000000 P 32ac7f36a4444784a5e84c7f6304c127 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:04:41.171875 18495 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32ac7f36a4444784a5e84c7f6304c127 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32ac7f36a4444784a5e84c7f6304c127, State: Initialized, Role: FOLLOWER
I20260806 03:04:41.172441 18495 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32ac7f36a4444784a5e84c7f6304c127 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32ac7f36a4444784a5e84c7f6304c127" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 40781 } }
I20260806 03:04:41.172660 18495 raft_consensus.cc:388] T 00000000000000000000000000000000 P 32ac7f36a4444784a5e84c7f6304c127 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:04:41.172839 18495 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32ac7f36a4444784a5e84c7f6304c127 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:04:41.173024 18495 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32ac7f36a4444784a5e84c7f6304c127 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:04:41.175710 18495 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32ac7f36a4444784a5e84c7f6304c127 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32ac7f36a4444784a5e84c7f6304c127" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 40781 } }
I20260806 03:04:41.176278 18495 leader_election.cc:302] T 00000000000000000000000000000000 P 32ac7f36a4444784a5e84c7f6304c127 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32ac7f36a4444784a5e84c7f6304c127; no voters:
I20260806 03:04:41.176944 18495 leader_election.cc:288] T 00000000000000000000000000000000 P 32ac7f36a4444784a5e84c7f6304c127 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:04:41.177117 18500 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32ac7f36a4444784a5e84c7f6304c127 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:04:41.177939 18500 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32ac7f36a4444784a5e84c7f6304c127 [term 1 LEADER]: Becoming Leader. State: Replica: 32ac7f36a4444784a5e84c7f6304c127, State: Running, Role: LEADER
I20260806 03:04:41.179178 18495 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32ac7f36a4444784a5e84c7f6304c127 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:04:41.179041 18500 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32ac7f36a4444784a5e84c7f6304c127 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32ac7f36a4444784a5e84c7f6304c127" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 40781 } }
I20260806 03:04:41.185459 18501 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32ac7f36a4444784a5e84c7f6304c127 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "32ac7f36a4444784a5e84c7f6304c127" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32ac7f36a4444784a5e84c7f6304c127" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 40781 } } }
I20260806 03:04:41.185410 18502 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32ac7f36a4444784a5e84c7f6304c127 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32ac7f36a4444784a5e84c7f6304c127. Latest consensus state: current_term: 1 leader_uuid: "32ac7f36a4444784a5e84c7f6304c127" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32ac7f36a4444784a5e84c7f6304c127" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 40781 } } }
I20260806 03:04:41.187171 18502 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32ac7f36a4444784a5e84c7f6304c127 [sys.catalog]: This master's current role is: LEADER
I20260806 03:04:41.187171 18501 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32ac7f36a4444784a5e84c7f6304c127 [sys.catalog]: This master's current role is: LEADER
I20260806 03:04:41.195739 18511 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:04:41.205389 18511 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:04:41.219542 18511 catalog_manager.cc:1347] Generated new cluster ID: 846cbd5b8a154db28769fef929ca14e2
I20260806 03:04:41.219672 18511 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:04:41.235090 18511 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:04:41.236600 18511 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:04:41.243866 18511 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 32ac7f36a4444784a5e84c7f6304c127: Generated new TSK 0
I20260806 03:04:41.245210 18511 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.74.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.193.keytab.
Entry for principal kudu/127.31.74.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.193.keytab.
I20260806 03:04:41.353092 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.193:0
--local_ip_for_outbound_sockets=127.31.74.193
--webserver_interface=127.31.74.193
--webserver_port=0
--tserver_master_addrs=127.31.74.254:40781
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.193.keytab
--principal=kudu/127.31.74.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:39993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:04:41.780850 18521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:04:41.781165 18521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:04:41.781337 18521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:04:41.789484 18521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:04:41.789623 18521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:04:41.789813 18521 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.193
I20260806 03:04:41.801136 18521 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:39993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.193.keytab
--principal=kudu/127.31.74.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.74.193
--webserver_port=0
--tserver_master_addrs=127.31.74.254:40781
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.193
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:04:41.802820 18521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:04:41.804695 18521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:04:41.820241 18526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:04:41.820163 18527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:04:41.820215 18529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:04:41.821584 18521 server_base.cc:1034] running on GCE node
I20260806 03:04:41.822355 18521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:04:41.823485 18521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:04:41.824819 18521 hybrid_clock.cc:648] HybridClock initialized: now 1785985481824741 us; error 69 us; skew 500 ppm
I20260806 03:04:41.825277 18521 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 03:04:41 dist-test-slave-dhph krb5kdc[18410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985481, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:04:41.829933 18521 init.cc:377] Logged in from keytab as kudu/127.31.74.193@KRBTEST.COM (short username kudu)
I20260806 03:04:41.831979 18521 webserver.cc:460] Webserver started at http://127.31.74.193:37339/ using document root <none> and password file <none>
I20260806 03:04:41.832774 18521 fs_manager.cc:359] Metadata directory not provided
I20260806 03:04:41.832942 18521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:04:41.833292 18521 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:04:41.836514 18521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/minicluster-data/ts-0/data/instance:
uuid: "f022f0941a3848cb89c09901b649f698"
format_stamp: "Formatted at 2026-08-06 03:04:41 on dist-test-slave-dhph"
server_key: "72316c7a8989299091012cb758a17738"
server_key_iv: "f0a40a9ba0f2008c62cc69e30d542ffe"
server_key_version: "encryptionkey@0"
I20260806 03:04:41.837370 18521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/minicluster-data/ts-0/wal/instance:
uuid: "f022f0941a3848cb89c09901b649f698"
format_stamp: "Formatted at 2026-08-06 03:04:41 on dist-test-slave-dhph"
server_key: "72316c7a8989299091012cb758a17738"
server_key_iv: "f0a40a9ba0f2008c62cc69e30d542ffe"
server_key_version: "encryptionkey@0"
I20260806 03:04:41.843329 18521 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260806 03:04:41.847733 18536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:04:41.849810 18521 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260806 03:04:41.850132 18521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/minicluster-data/ts-0/wal
uuid: "f022f0941a3848cb89c09901b649f698"
format_stamp: "Formatted at 2026-08-06 03:04:41 on dist-test-slave-dhph"
server_key: "72316c7a8989299091012cb758a17738"
server_key_iv: "f0a40a9ba0f2008c62cc69e30d542ffe"
server_key_version: "encryptionkey@0"
I20260806 03:04:41.850384 18521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:04:41.881971 18521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:04:41.887890 18521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:04:41.888286 18521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:04:41.889531 18521 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:04:41.891212 18521 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:04:41.891330 18521 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:04:41.891492 18521 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:04:41.891569 18521 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:04:41.935369 18521 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.193:39865
I20260806 03:04:41.935468 18654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.193:39865 every 8 connection(s)
I20260806 03:04:41.936800 18521 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
I20260806 03:04:41.946480 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 18521
I20260806 03:04:41.946771 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/minicluster-data/ts-0/wal/instance
I20260806 03:04:41.947468 32043 external_mini_cluster.cc:1428] Setting key 581b4650a3a303babb2b069d728b5d12
Aug 06 03:04:41 dist-test-slave-dhph krb5kdc[18410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985481, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.193@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 03:04:41.962936 18657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:41.939647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.193:60301 (local address 127.31.74.254:40781)
0806 03:04:41.940170 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:41.940177 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:41.941334 (+ 1157us) server_negotiation.cc:407] Connection header received
0806 03:04:41.943956 (+ 2622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:41.943961 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:41.944046 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:41.944157 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:04:41.946141 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:41.947256 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:41.949905 (+ 2649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:41.950411 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:41.950518 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:41.955716 (+ 5198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:04:41.955723 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:04:41.955726 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:04:41.955770 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:04:41.958972 (+ 3202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:04:41.959927 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:04:41.959934 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:04:41.959938 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:04:41.960084 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:04:41.960568 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:04:41.960574 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:04:41.960578 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:04:41.960894 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:04:41.961071 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:41.961890 (+ 819us) server_negotiation.cc:299] Negotiation successful
0806 03:04:41.962481 (+ 591us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":265,"threads_started":1}
I20260806 03:04:41.964073 18658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:41.940312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:40781 (local address 127.31.74.193:60301)
0806 03:04:41.941141 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:41.941192 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 03:04:41.943269 (+ 2077us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:04:41.944477 (+ 1208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:04:41.944487 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:04:41.945114 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:04:41.945835 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:41.945864 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:41.947484 (+ 1620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:41.947493 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:41.949625 (+ 2132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:41.949645 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:41.950715 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:41.950723 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:41.951053 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:41.951961 (+ 908us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:04:41.952001 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:04:41.955376 (+ 3375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:04:41.959200 (+ 3824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:04:41.959214 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:04:41.959244 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:04:41.959783 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:04:41.960205 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:04:41.960210 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:04:41.960214 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:04:41.960438 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:04:41.961052 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:04:41.961061 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:04:41.961480 (+ 419us) client_negotiation.cc:769] Sending connection context
0806 03:04:41.961773 (+ 293us) client_negotiation.cc:240] Negotiation successful
0806 03:04:41.962859 (+ 1086us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":349,"threads_started":1}
I20260806 03:04:41.966032 18655 heartbeater.cc:347] Connected to a master server at 127.31.74.254:40781
I20260806 03:04:41.966452 18655 heartbeater.cc:464] Registering TS with master...
I20260806 03:04:41.967458 18655 heartbeater.cc:511] Master 127.31.74.254:40781 requested a full tablet report, sending...
I20260806 03:04:41.970274 18458 ts_manager.cc:194] Registered new tserver with Master: f022f0941a3848cb89c09901b649f698 (127.31.74.193:39865)
I20260806 03:04:41.971964 18458 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.74.193@KRBTEST.COM'} at 127.31.74.193:60301
WARNING: no policy specified for kudu/127.31.74.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.194.keytab.
Entry for principal kudu/127.31.74.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.194.keytab.
I20260806 03:04:42.037004 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.194:0
--local_ip_for_outbound_sockets=127.31.74.194
--webserver_interface=127.31.74.194
--webserver_port=0
--tserver_master_addrs=127.31.74.254:40781
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.194.keytab
--principal=kudu/127.31.74.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:39993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:04:42.457499 18661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:04:42.457906 18661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:04:42.458091 18661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:04:42.467403 18661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:04:42.467600 18661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:04:42.467788 18661 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.194
I20260806 03:04:42.480342 18661 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:39993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.194.keytab
--principal=kudu/127.31.74.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.74.194
--webserver_port=0
--tserver_master_addrs=127.31.74.254:40781
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.194
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:04:42.481935 18661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:04:42.483467 18661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:04:42.499473 18666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:04:42.499562 18667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:04:42.499830 18669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:04:42.500960 18661 server_base.cc:1034] running on GCE node
I20260806 03:04:42.501709 18661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:04:42.502923 18661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:04:42.504281 18661 hybrid_clock.cc:648] HybridClock initialized: now 1785985482504186 us; error 52 us; skew 500 ppm
I20260806 03:04:42.505067 18661 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 03:04:42 dist-test-slave-dhph krb5kdc[18410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985482, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:04:42.509310 18661 init.cc:377] Logged in from keytab as kudu/127.31.74.194@KRBTEST.COM (short username kudu)
I20260806 03:04:42.511979 18661 webserver.cc:460] Webserver started at http://127.31.74.194:43663/ using document root <none> and password file <none>
I20260806 03:04:42.512768 18661 fs_manager.cc:359] Metadata directory not provided
I20260806 03:04:42.512888 18661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:04:42.513173 18661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:04:42.516396 18661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/minicluster-data/ts-1/data/instance:
uuid: "2bfd1fa177b2471ca35293bbab14f8d0"
format_stamp: "Formatted at 2026-08-06 03:04:42 on dist-test-slave-dhph"
server_key: "f421fb4b8c534b967d07f15ee20667c2"
server_key_iv: "1d34fac53d9477c9e36b995e0440550b"
server_key_version: "encryptionkey@0"
I20260806 03:04:42.517192 18661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/minicluster-data/ts-1/wal/instance:
uuid: "2bfd1fa177b2471ca35293bbab14f8d0"
format_stamp: "Formatted at 2026-08-06 03:04:42 on dist-test-slave-dhph"
server_key: "f421fb4b8c534b967d07f15ee20667c2"
server_key_iv: "1d34fac53d9477c9e36b995e0440550b"
server_key_version: "encryptionkey@0"
I20260806 03:04:42.523411 18661 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260806 03:04:42.527945 18676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:04:42.529727 18661 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260806 03:04:42.529958 18661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/minicluster-data/ts-1/wal
uuid: "2bfd1fa177b2471ca35293bbab14f8d0"
format_stamp: "Formatted at 2026-08-06 03:04:42 on dist-test-slave-dhph"
server_key: "f421fb4b8c534b967d07f15ee20667c2"
server_key_iv: "1d34fac53d9477c9e36b995e0440550b"
server_key_version: "encryptionkey@0"
I20260806 03:04:42.530243 18661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:04:42.550555 18661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:04:42.556435 18661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:04:42.556780 18661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:04:42.558025 18661 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:04:42.559551 18661 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:04:42.559628 18661 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:04:42.559756 18661 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:04:42.559834 18661 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:04:42.605918 18661 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.194:40771
I20260806 03:04:42.606019 18791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.194:40771 every 8 connection(s)
I20260806 03:04:42.607617 18661 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
I20260806 03:04:42.608861 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 18661
I20260806 03:04:42.609194 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/minicluster-data/ts-1/wal/instance
I20260806 03:04:42.609834 32043 external_mini_cluster.cc:1428] Setting key de0bd161a67961bc572ddb74c82c4de8
Aug 06 03:04:42 dist-test-slave-dhph krb5kdc[18410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985482, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.194@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 03:04:42.634985 18794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:42.610862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.194:44255 (local address 127.31.74.254:40781)
0806 03:04:42.611365 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:42.611371 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:42.612746 (+ 1375us) server_negotiation.cc:407] Connection header received
0806 03:04:42.614685 (+ 1939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:42.614693 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:42.614796 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:42.614924 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:04:42.616887 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:42.618053 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:42.620358 (+ 2305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:42.621410 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:42.621575 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:42.626820 (+ 5245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:04:42.626831 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:04:42.626839 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:04:42.626897 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:04:42.630813 (+ 3916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:04:42.632127 (+ 1314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:04:42.632131 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:04:42.632133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:04:42.632279 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:04:42.632825 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:04:42.632834 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:04:42.632836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:04:42.633144 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:04:42.633323 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:42.634189 (+ 866us) server_negotiation.cc:299] Negotiation successful
0806 03:04:42.634441 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":253,"threads_started":1}
I20260806 03:04:42.636010 18795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:42.611419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:40781 (local address 127.31.74.194:44255)
0806 03:04:42.612564 (+ 1145us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:42.612616 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 03:04:42.614375 (+ 1759us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:04:42.615186 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:04:42.615217 (+ 31us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:04:42.616007 (+ 790us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:04:42.616661 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:42.616691 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:42.618249 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:42.618253 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:42.620174 (+ 1921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:42.620195 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:42.621672 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:42.621681 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:42.622022 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:42.622913 (+ 891us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:04:42.622948 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:04:42.626390 (+ 3442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:04:42.631021 (+ 4631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:04:42.631036 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:04:42.631061 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:04:42.631880 (+ 819us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:04:42.632454 (+ 574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:04:42.632458 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:04:42.632460 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:04:42.632676 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:04:42.633302 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:04:42.633312 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:04:42.633704 (+ 392us) client_negotiation.cc:769] Sending connection context
0806 03:04:42.634095 (+ 391us) client_negotiation.cc:240] Negotiation successful
0806 03:04:42.634742 (+ 647us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"thread_start_us":404,"threads_started":1}
I20260806 03:04:42.637889 18792 heartbeater.cc:347] Connected to a master server at 127.31.74.254:40781
I20260806 03:04:42.638260 18792 heartbeater.cc:464] Registering TS with master...
I20260806 03:04:42.639358 18792 heartbeater.cc:511] Master 127.31.74.254:40781 requested a full tablet report, sending...
I20260806 03:04:42.641266 18458 ts_manager.cc:194] Registered new tserver with Master: 2bfd1fa177b2471ca35293bbab14f8d0 (127.31.74.194:40771)
I20260806 03:04:42.642246 18458 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.74.194@KRBTEST.COM'} at 127.31.74.194:44255
WARNING: no policy specified for kudu/127.31.74.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.195.keytab.
Entry for principal kudu/127.31.74.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.195.keytab.
I20260806 03:04:42.699890 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.195:0
--local_ip_for_outbound_sockets=127.31.74.195
--webserver_interface=127.31.74.195
--webserver_port=0
--tserver_master_addrs=127.31.74.254:40781
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.195.keytab
--principal=kudu/127.31.74.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:39993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 03:04:42.976486 18655 heartbeater.cc:503] Master 127.31.74.254:40781 was elected leader, sending a full tablet report...
W20260806 03:04:43.126025 18798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:04:43.126461 18798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:04:43.126726 18798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:04:43.137436 18798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:04:43.137687 18798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:04:43.137907 18798 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.195
I20260806 03:04:43.153357 18798 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:39993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.195.keytab
--principal=kudu/127.31.74.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.74.195
--webserver_port=0
--tserver_master_addrs=127.31.74.254:40781
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.195
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:04:43.155077 18798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:04:43.156527 18798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:04:43.172808 18803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:04:43.172855 18804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:04:43.172772 18806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:04:43.180081 18798 server_base.cc:1034] running on GCE node
I20260806 03:04:43.181340 18798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:04:43.182971 18798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:04:43.184403 18798 hybrid_clock.cc:648] HybridClock initialized: now 1785985483184308 us; error 102 us; skew 500 ppm
I20260806 03:04:43.184832 18798 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 03:04:43 dist-test-slave-dhph krb5kdc[18410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985483, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:04:43.188858 18798 init.cc:377] Logged in from keytab as kudu/127.31.74.195@KRBTEST.COM (short username kudu)
I20260806 03:04:43.191540 18798 webserver.cc:460] Webserver started at http://127.31.74.195:41083/ using document root <none> and password file <none>
I20260806 03:04:43.193085 18798 fs_manager.cc:359] Metadata directory not provided
I20260806 03:04:43.193272 18798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:04:43.193655 18798 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:04:43.196952 18798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/minicluster-data/ts-2/data/instance:
uuid: "5a3b236510d04c18a3b79f9a59528115"
format_stamp: "Formatted at 2026-08-06 03:04:43 on dist-test-slave-dhph"
server_key: "4d5147b08e762f5f20a1492c19152991"
server_key_iv: "3f4666c53dacd22d42b353bf2db00750"
server_key_version: "encryptionkey@0"
I20260806 03:04:43.197850 18798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/minicluster-data/ts-2/wal/instance:
uuid: "5a3b236510d04c18a3b79f9a59528115"
format_stamp: "Formatted at 2026-08-06 03:04:43 on dist-test-slave-dhph"
server_key: "4d5147b08e762f5f20a1492c19152991"
server_key_iv: "3f4666c53dacd22d42b353bf2db00750"
server_key_version: "encryptionkey@0"
I20260806 03:04:43.204401 18798 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.008s
I20260806 03:04:43.209048 18813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:04:43.210850 18798 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 03:04:43.211073 18798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/minicluster-data/ts-2/wal
uuid: "5a3b236510d04c18a3b79f9a59528115"
format_stamp: "Formatted at 2026-08-06 03:04:43 on dist-test-slave-dhph"
server_key: "4d5147b08e762f5f20a1492c19152991"
server_key_iv: "3f4666c53dacd22d42b353bf2db00750"
server_key_version: "encryptionkey@0"
I20260806 03:04:43.211362 18798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:04:43.238217 18798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:04:43.243973 18798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:04:43.244436 18798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:04:43.245710 18798 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:04:43.248143 18798 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:04:43.248253 18798 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:04:43.248414 18798 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:04:43.248492 18798 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:04:43.292335 18798 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.195:40043
I20260806 03:04:43.292393 18928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.195:40043 every 8 connection(s)
I20260806 03:04:43.294009 18798 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
I20260806 03:04:43.304323 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 18798
I20260806 03:04:43.304559 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/minicluster-data/ts-2/wal/instance
I20260806 03:04:43.305199 32043 external_mini_cluster.cc:1428] Setting key 677b6d9aa45c05750a8b6306333f03bb
Aug 06 03:04:43 dist-test-slave-dhph krb5kdc[18410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985483, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.195@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 03:04:43.320784 18931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:43.296907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.195:49497 (local address 127.31.74.254:40781)
0806 03:04:43.297493 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:43.297500 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:43.298924 (+ 1424us) server_negotiation.cc:407] Connection header received
0806 03:04:43.301363 (+ 2439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:43.301368 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:43.301475 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:43.301605 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:04:43.303644 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:43.305080 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:43.307413 (+ 2333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:43.308027 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:43.308182 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:43.313059 (+ 4877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:04:43.313066 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:04:43.313070 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:04:43.313113 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:04:43.316429 (+ 3316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:04:43.317497 (+ 1068us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:04:43.317502 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:04:43.317505 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:04:43.317657 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:04:43.318260 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:04:43.318264 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:04:43.318266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:04:43.318676 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:04:43.318838 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:43.319885 (+ 1047us) server_negotiation.cc:299] Negotiation successful
0806 03:04:43.320144 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":288,"threads_started":1}
I20260806 03:04:43.322393 18932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:43.297716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:40781 (local address 127.31.74.195:49497)
0806 03:04:43.298534 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:43.298778 (+ 244us) client_negotiation.cc:174] Beginning negotiation
0806 03:04:43.300709 (+ 1931us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:04:43.301992 (+ 1283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:04:43.302004 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:04:43.302731 (+ 727us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:04:43.303401 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:43.303422 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:43.305344 (+ 1922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:43.305351 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:43.307235 (+ 1884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:43.307250 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:43.308573 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:43.308585 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:43.308836 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:43.309835 (+ 999us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:04:43.309873 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:04:43.312682 (+ 2809us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:04:43.316667 (+ 3985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:04:43.316675 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:04:43.316690 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:04:43.317313 (+ 623us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:04:43.317813 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:04:43.317817 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:04:43.317820 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:04:43.318075 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:04:43.318863 (+ 788us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:04:43.318872 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:04:43.319345 (+ 473us) client_negotiation.cc:769] Sending connection context
0806 03:04:43.319679 (+ 334us) client_negotiation.cc:240] Negotiation successful
0806 03:04:43.320503 (+ 824us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":318,"threads_started":1}
I20260806 03:04:43.324055 18929 heartbeater.cc:347] Connected to a master server at 127.31.74.254:40781
I20260806 03:04:43.324445 18929 heartbeater.cc:464] Registering TS with master...
I20260806 03:04:43.325274 18929 heartbeater.cc:511] Master 127.31.74.254:40781 requested a full tablet report, sending...
I20260806 03:04:43.327193 18458 ts_manager.cc:194] Registered new tserver with Master: 5a3b236510d04c18a3b79f9a59528115 (127.31.74.195:40043)
I20260806 03:04:43.328555 18458 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.74.195@KRBTEST.COM'} at 127.31.74.195:49497
I20260806 03:04:43.336596 32043 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 03:04:43.357599 18931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:43.341695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45148 (local address 127.31.74.254:40781)
0806 03:04:43.342014 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:43.342022 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:43.342387 (+ 365us) server_negotiation.cc:407] Connection header received
0806 03:04:43.342660 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:43.342665 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:43.342733 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:43.342855 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:04:43.344369 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:43.345132 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:43.346757 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:43.347234 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:43.347434 (+ 200us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:43.350642 (+ 3208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:04:43.350650 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:04:43.350656 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:04:43.350709 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:04:43.354009 (+ 3300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:04:43.355195 (+ 1186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:04:43.355204 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:04:43.355213 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:04:43.355347 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:04:43.355886 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:04:43.355894 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:04:43.355902 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:04:43.356249 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:04:43.356554 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:43.357029 (+ 475us) server_negotiation.cc:299] Negotiation successful
0806 03:04:43.357269 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145}
I20260806 03:04:43.364203 18458 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45148:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 03:04:43.371755 18458 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 03:04:43.423035 18946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:43.388044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:39865 (local address 127.0.0.1:45350)
0806 03:04:43.397873 (+ 9829us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:43.397954 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0806 03:04:43.398224 (+ 270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:04:43.412937 (+ 14713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:04:43.412941 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:04:43.412971 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:04:43.413685 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:43.413703 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:43.415234 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:43.415240 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:43.418761 (+ 3521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:43.418791 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:43.421170 (+ 2379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:43.421179 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:43.421476 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:43.421547 (+ 71us) client_negotiation.cc:769] Sending connection context
0806 03:04:43.421741 (+ 194us) client_negotiation.cc:240] Negotiation successful
0806 03:04:43.422168 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9441,"mutex_wait_us":685,"spinlock_wait_cycles":11008,"thread_start_us":22765,"threads_started":1}
I20260806 03:04:43.423094 18947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:43.395130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52956 (local address 127.31.74.195:40043)
0806 03:04:43.409742 (+ 14612us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:43.409759 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:43.411928 (+ 2169us) server_negotiation.cc:407] Connection header received
0806 03:04:43.412296 (+ 368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:43.412302 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:43.412546 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:43.412731 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:04:43.413993 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:43.415562 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:43.419514 (+ 3952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:43.420779 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:43.420878 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:43.421171 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:43.421842 (+ 671us) server_negotiation.cc:299] Negotiation successful
0806 03:04:43.422353 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14287,"thread_start_us":14659,"threads_started":1}
I20260806 03:04:43.423597 18945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:43.389049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47484 (local address 127.31.74.194:40771)
0806 03:04:43.410756 (+ 21707us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:43.410772 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:43.411487 (+ 715us) server_negotiation.cc:407] Connection header received
0806 03:04:43.411853 (+ 366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:43.411865 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:43.412213 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:43.412453 (+ 240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:04:43.413504 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:43.415180 (+ 1676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:43.419842 (+ 4662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:43.421494 (+ 1652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:43.421695 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:43.421899 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:43.422638 (+ 739us) server_negotiation.cc:299] Negotiation successful
0806 03:04:43.422894 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21378,"thread_start_us":21237,"threads_started":1}
I20260806 03:04:43.424078 18943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:43.394343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.195:40043 (local address 127.0.0.1:52956)
0806 03:04:43.411734 (+ 17391us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:43.411769 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0806 03:04:43.412070 (+ 301us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:04:43.413131 (+ 1061us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:04:43.413141 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:04:43.413335 (+ 194us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:04:43.413783 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:43.413793 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:43.415741 (+ 1948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:43.415747 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:43.419249 (+ 3502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:43.419271 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:43.421378 (+ 2107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:43.421384 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:43.421538 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:43.421565 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 03:04:43.423395 (+ 1830us) client_negotiation.cc:240] Negotiation successful
0806 03:04:43.423586 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17218,"mutex_wait_us":67,"spinlock_wait_cycles":1280,"thread_start_us":2936,"threads_started":1}
I20260806 03:04:43.424523 18942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:43.388405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.194:40771 (local address 127.0.0.1:47484)
0806 03:04:43.411327 (+ 22922us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:43.411364 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 03:04:43.411625 (+ 261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:04:43.412496 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:04:43.412504 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:04:43.412548 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:04:43.413120 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:43.413133 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:43.415365 (+ 2232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:43.415371 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:43.419574 (+ 4203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:43.419593 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:43.421804 (+ 2211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:43.421813 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:43.422341 (+ 528us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:43.422372 (+ 31us) client_negotiation.cc:769] Sending connection context
0806 03:04:43.422509 (+ 137us) client_negotiation.cc:240] Negotiation successful
0806 03:04:43.423870 (+ 1361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22698,"mutex_wait_us":606,"thread_start_us":22943,"threads_started":1}
I20260806 03:04:43.424926 18944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:43.388391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45350 (local address 127.31.74.193:39865)
0806 03:04:43.412361 (+ 23970us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:43.412371 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:43.412400 (+ 29us) server_negotiation.cc:407] Connection header received
0806 03:04:43.412494 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:43.412501 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:43.412722 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:43.412936 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:04:43.413912 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:43.415057 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:43.419066 (+ 4009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:43.420841 (+ 1775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:43.421069 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:43.421388 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:43.424266 (+ 2878us) server_negotiation.cc:299] Negotiation successful
0806 03:04:43.424527 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23758,"thread_start_us":23541,"threads_started":1}
I20260806 03:04:43.428900 18861 tablet_service.cc:1467] Processing CreateTablet for tablet c085eba91aa24acebf42d525e95cd184 (DEFAULT_TABLE table=test-table [id=83b3f671a7db40c684c12c58ca5c1e6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:04:43.429287 18587 tablet_service.cc:1467] Processing CreateTablet for tablet c085eba91aa24acebf42d525e95cd184 (DEFAULT_TABLE table=test-table [id=83b3f671a7db40c684c12c58ca5c1e6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:04:43.429277 18724 tablet_service.cc:1467] Processing CreateTablet for tablet c085eba91aa24acebf42d525e95cd184 (DEFAULT_TABLE table=test-table [id=83b3f671a7db40c684c12c58ca5c1e6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:04:43.430831 18587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c085eba91aa24acebf42d525e95cd184. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:04:43.430902 18724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c085eba91aa24acebf42d525e95cd184. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:04:43.431022 18861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c085eba91aa24acebf42d525e95cd184. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:04:43.447489 18948 tablet_bootstrap.cc:492] T c085eba91aa24acebf42d525e95cd184 P 2bfd1fa177b2471ca35293bbab14f8d0: Bootstrap starting.
I20260806 03:04:43.451436 18950 tablet_bootstrap.cc:492] T c085eba91aa24acebf42d525e95cd184 P f022f0941a3848cb89c09901b649f698: Bootstrap starting.
I20260806 03:04:43.452241 18948 tablet_bootstrap.cc:654] T c085eba91aa24acebf42d525e95cd184 P 2bfd1fa177b2471ca35293bbab14f8d0: Neither blocks nor log segments found. Creating new log.
I20260806 03:04:43.454052 18948 log.cc:824] T c085eba91aa24acebf42d525e95cd184 P 2bfd1fa177b2471ca35293bbab14f8d0: Log is configured to *not* fsync() on all Append() calls
I20260806 03:04:43.454527 18949 tablet_bootstrap.cc:492] T c085eba91aa24acebf42d525e95cd184 P 5a3b236510d04c18a3b79f9a59528115: Bootstrap starting.
I20260806 03:04:43.456748 18950 tablet_bootstrap.cc:654] T c085eba91aa24acebf42d525e95cd184 P f022f0941a3848cb89c09901b649f698: Neither blocks nor log segments found. Creating new log.
I20260806 03:04:43.457856 18950 log.cc:824] T c085eba91aa24acebf42d525e95cd184 P f022f0941a3848cb89c09901b649f698: Log is configured to *not* fsync() on all Append() calls
I20260806 03:04:43.459245 18949 tablet_bootstrap.cc:654] T c085eba91aa24acebf42d525e95cd184 P 5a3b236510d04c18a3b79f9a59528115: Neither blocks nor log segments found. Creating new log.
I20260806 03:04:43.460579 18949 log.cc:824] T c085eba91aa24acebf42d525e95cd184 P 5a3b236510d04c18a3b79f9a59528115: Log is configured to *not* fsync() on all Append() calls
I20260806 03:04:43.461758 18948 tablet_bootstrap.cc:492] T c085eba91aa24acebf42d525e95cd184 P 2bfd1fa177b2471ca35293bbab14f8d0: No bootstrap required, opened a new log
I20260806 03:04:43.461756 18950 tablet_bootstrap.cc:492] T c085eba91aa24acebf42d525e95cd184 P f022f0941a3848cb89c09901b649f698: No bootstrap required, opened a new log
I20260806 03:04:43.462073 18948 ts_tablet_manager.cc:1397] T c085eba91aa24acebf42d525e95cd184 P 2bfd1fa177b2471ca35293bbab14f8d0: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.006s
I20260806 03:04:43.462073 18950 ts_tablet_manager.cc:1397] T c085eba91aa24acebf42d525e95cd184 P f022f0941a3848cb89c09901b649f698: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260806 03:04:43.463323 18949 tablet_bootstrap.cc:492] T c085eba91aa24acebf42d525e95cd184 P 5a3b236510d04c18a3b79f9a59528115: No bootstrap required, opened a new log
I20260806 03:04:43.463601 18949 ts_tablet_manager.cc:1397] T c085eba91aa24acebf42d525e95cd184 P 5a3b236510d04c18a3b79f9a59528115: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260806 03:04:43.471246 18949 raft_consensus.cc:348] T c085eba91aa24acebf42d525e95cd184 P 5a3b236510d04c18a3b79f9a59528115 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f022f0941a3848cb89c09901b649f698" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 39865 } } peers { permanent_uuid: "2bfd1fa177b2471ca35293bbab14f8d0" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 40771 } } peers { permanent_uuid: "5a3b236510d04c18a3b79f9a59528115" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 40043 } }
I20260806 03:04:43.471699 18949 raft_consensus.cc:374] T c085eba91aa24acebf42d525e95cd184 P 5a3b236510d04c18a3b79f9a59528115 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:04:43.471884 18949 raft_consensus.cc:729] T c085eba91aa24acebf42d525e95cd184 P 5a3b236510d04c18a3b79f9a59528115 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a3b236510d04c18a3b79f9a59528115, State: Initialized, Role: FOLLOWER
I20260806 03:04:43.472572 18949 consensus_queue.cc:260] T c085eba91aa24acebf42d525e95cd184 P 5a3b236510d04c18a3b79f9a59528115 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f022f0941a3848cb89c09901b649f698" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 39865 } } peers { permanent_uuid: "2bfd1fa177b2471ca35293bbab14f8d0" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 40771 } } peers { permanent_uuid: "5a3b236510d04c18a3b79f9a59528115" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 40043 } }
I20260806 03:04:43.473809 18929 heartbeater.cc:503] Master 127.31.74.254:40781 was elected leader, sending a full tablet report...
I20260806 03:04:43.474617 18949 ts_tablet_manager.cc:1428] T c085eba91aa24acebf42d525e95cd184 P 5a3b236510d04c18a3b79f9a59528115: Time spent starting tablet: real 0.011s user 0.007s sys 0.004s
I20260806 03:04:43.475119 18948 raft_consensus.cc:348] T c085eba91aa24acebf42d525e95cd184 P 2bfd1fa177b2471ca35293bbab14f8d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f022f0941a3848cb89c09901b649f698" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 39865 } } peers { permanent_uuid: "2bfd1fa177b2471ca35293bbab14f8d0" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 40771 } } peers { permanent_uuid: "5a3b236510d04c18a3b79f9a59528115" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 40043 } }
I20260806 03:04:43.475119 18950 raft_consensus.cc:348] T c085eba91aa24acebf42d525e95cd184 P f022f0941a3848cb89c09901b649f698 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f022f0941a3848cb89c09901b649f698" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 39865 } } peers { permanent_uuid: "2bfd1fa177b2471ca35293bbab14f8d0" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 40771 } } peers { permanent_uuid: "5a3b236510d04c18a3b79f9a59528115" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 40043 } }
I20260806 03:04:43.475629 18950 raft_consensus.cc:374] T c085eba91aa24acebf42d525e95cd184 P f022f0941a3848cb89c09901b649f698 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:04:43.475730 18948 raft_consensus.cc:374] T c085eba91aa24acebf42d525e95cd184 P 2bfd1fa177b2471ca35293bbab14f8d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:04:43.475808 18950 raft_consensus.cc:729] T c085eba91aa24acebf42d525e95cd184 P f022f0941a3848cb89c09901b649f698 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f022f0941a3848cb89c09901b649f698, State: Initialized, Role: FOLLOWER
I20260806 03:04:43.475890 18948 raft_consensus.cc:729] T c085eba91aa24acebf42d525e95cd184 P 2bfd1fa177b2471ca35293bbab14f8d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bfd1fa177b2471ca35293bbab14f8d0, State: Initialized, Role: FOLLOWER
I20260806 03:04:43.476645 18948 consensus_queue.cc:260] T c085eba91aa24acebf42d525e95cd184 P 2bfd1fa177b2471ca35293bbab14f8d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f022f0941a3848cb89c09901b649f698" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 39865 } } peers { permanent_uuid: "2bfd1fa177b2471ca35293bbab14f8d0" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 40771 } } peers { permanent_uuid: "5a3b236510d04c18a3b79f9a59528115" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 40043 } }
I20260806 03:04:43.476645 18950 consensus_queue.cc:260] T c085eba91aa24acebf42d525e95cd184 P f022f0941a3848cb89c09901b649f698 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f022f0941a3848cb89c09901b649f698" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 39865 } } peers { permanent_uuid: "2bfd1fa177b2471ca35293bbab14f8d0" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 40771 } } peers { permanent_uuid: "5a3b236510d04c18a3b79f9a59528115" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 40043 } }
I20260806 03:04:43.478391 18792 heartbeater.cc:503] Master 127.31.74.254:40781 was elected leader, sending a full tablet report...
I20260806 03:04:43.479311 18948 ts_tablet_manager.cc:1428] T c085eba91aa24acebf42d525e95cd184 P 2bfd1fa177b2471ca35293bbab14f8d0: Time spent starting tablet: real 0.017s user 0.015s sys 0.004s
I20260806 03:04:43.481319 18950 ts_tablet_manager.cc:1428] T c085eba91aa24acebf42d525e95cd184 P f022f0941a3848cb89c09901b649f698: Time spent starting tablet: real 0.019s user 0.014s sys 0.004s
W20260806 03:04:43.546029 18930 tablet.cc:2369] T c085eba91aa24acebf42d525e95cd184 P 5a3b236510d04c18a3b79f9a59528115: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 03:04:43.610440 18793 tablet.cc:2369] T c085eba91aa24acebf42d525e95cd184 P 2bfd1fa177b2471ca35293bbab14f8d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 03:04:43.640442 18956 raft_consensus.cc:482] T c085eba91aa24acebf42d525e95cd184 P f022f0941a3848cb89c09901b649f698 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:04:43.640818 18956 raft_consensus.cc:504] T c085eba91aa24acebf42d525e95cd184 P f022f0941a3848cb89c09901b649f698 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f022f0941a3848cb89c09901b649f698" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 39865 } } peers { permanent_uuid: "2bfd1fa177b2471ca35293bbab14f8d0" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 40771 } } peers { permanent_uuid: "5a3b236510d04c18a3b79f9a59528115" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 40043 } }
I20260806 03:04:43.642704 18956 leader_election.cc:288] T c085eba91aa24acebf42d525e95cd184 P f022f0941a3848cb89c09901b649f698 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2bfd1fa177b2471ca35293bbab14f8d0 (127.31.74.194:40771), 5a3b236510d04c18a3b79f9a59528115 (127.31.74.195:40043)
I20260806 03:04:43.652010 18945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:43.643499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.193:56885 (local address 127.31.74.194:40771)
0806 03:04:43.643889 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:43.643899 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:43.644143 (+ 244us) server_negotiation.cc:407] Connection header received
0806 03:04:43.644398 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:43.644403 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:43.644465 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:43.644566 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:04:43.645277 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:43.646448 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:43.649200 (+ 2752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:43.650813 (+ 1613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:43.650980 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:43.651055 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:43.651625 (+ 570us) server_negotiation.cc:299] Negotiation successful
0806 03:04:43.651761 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260806 03:04:43.652168 18957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:43.643263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.194:40771 (local address 127.31.74.193:56885)
0806 03:04:43.644018 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:43.644034 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 03:04:43.644237 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:04:43.644590 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:04:43.644594 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:04:43.644623 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:04:43.645085 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:43.645095 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:43.646705 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:43.646711 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:43.648954 (+ 2243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:43.648968 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:43.651059 (+ 2091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:43.651067 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:43.651268 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:43.651307 (+ 39us) client_negotiation.cc:769] Sending connection context
0806 03:04:43.651527 (+ 220us) client_negotiation.cc:240] Negotiation successful
0806 03:04:43.651693 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"mutex_wait_us":23,"thread_start_us":583,"threads_started":1}
I20260806 03:04:43.653439 18947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:43.643348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.193:59933 (local address 127.31.74.195:40043)
0806 03:04:43.643558 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:43.643567 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:43.644793 (+ 1226us) server_negotiation.cc:407] Connection header received
0806 03:04:43.645240 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:43.645245 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:43.645480 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:43.645573 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:04:43.646244 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:43.647822 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:43.650701 (+ 2879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:43.652147 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:43.652343 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:43.652432 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:43.652912 (+ 480us) server_negotiation.cc:299] Negotiation successful
0806 03:04:43.653054 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260806 03:04:43.653529 18958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:43.643485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.195:40043 (local address 127.31.74.193:59933)
0806 03:04:43.644680 (+ 1195us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:43.644707 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0806 03:04:43.645073 (+ 366us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:04:43.645636 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:04:43.645641 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:04:43.645668 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:04:43.646040 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:43.646051 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:43.648077 (+ 2026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:43.648082 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:43.650370 (+ 2288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:43.650391 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:43.652605 (+ 2214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:43.652610 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:43.652745 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:43.652769 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 03:04:43.652881 (+ 112us) client_negotiation.cc:240] Negotiation successful
0806 03:04:43.653021 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1071,"mutex_wait_us":33,"thread_start_us":860,"threads_started":1}
I20260806 03:04:43.653870 18745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c085eba91aa24acebf42d525e95cd184" candidate_uuid: "f022f0941a3848cb89c09901b649f698" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bfd1fa177b2471ca35293bbab14f8d0" is_pre_election: true
I20260806 03:04:43.654675 18745 raft_consensus.cc:2454] T c085eba91aa24acebf42d525e95cd184 P 2bfd1fa177b2471ca35293bbab14f8d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f022f0941a3848cb89c09901b649f698 in term 0.
I20260806 03:04:43.654809 18881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c085eba91aa24acebf42d525e95cd184" candidate_uuid: "f022f0941a3848cb89c09901b649f698" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a3b236510d04c18a3b79f9a59528115" is_pre_election: true
I20260806 03:04:43.655471 18881 raft_consensus.cc:2454] T c085eba91aa24acebf42d525e95cd184 P 5a3b236510d04c18a3b79f9a59528115 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f022f0941a3848cb89c09901b649f698 in term 0.
I20260806 03:04:43.655633 18538 leader_election.cc:302] T c085eba91aa24acebf42d525e95cd184 P f022f0941a3848cb89c09901b649f698 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bfd1fa177b2471ca35293bbab14f8d0, f022f0941a3848cb89c09901b649f698; no voters:
I20260806 03:04:43.656173 18956 raft_consensus.cc:2781] T c085eba91aa24acebf42d525e95cd184 P f022f0941a3848cb89c09901b649f698 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:04:43.656355 18956 raft_consensus.cc:482] T c085eba91aa24acebf42d525e95cd184 P f022f0941a3848cb89c09901b649f698 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:04:43.656504 18956 raft_consensus.cc:3037] T c085eba91aa24acebf42d525e95cd184 P f022f0941a3848cb89c09901b649f698 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:04:43.659540 18956 raft_consensus.cc:504] T c085eba91aa24acebf42d525e95cd184 P f022f0941a3848cb89c09901b649f698 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f022f0941a3848cb89c09901b649f698" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 39865 } } peers { permanent_uuid: "2bfd1fa177b2471ca35293bbab14f8d0" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 40771 } } peers { permanent_uuid: "5a3b236510d04c18a3b79f9a59528115" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 40043 } }
I20260806 03:04:43.660306 18956 leader_election.cc:288] T c085eba91aa24acebf42d525e95cd184 P f022f0941a3848cb89c09901b649f698 [CANDIDATE]: Term 1 election: Requested vote from peers 2bfd1fa177b2471ca35293bbab14f8d0 (127.31.74.194:40771), 5a3b236510d04c18a3b79f9a59528115 (127.31.74.195:40043)
I20260806 03:04:43.660905 18745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c085eba91aa24acebf42d525e95cd184" candidate_uuid: "f022f0941a3848cb89c09901b649f698" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bfd1fa177b2471ca35293bbab14f8d0"
I20260806 03:04:43.661110 18745 raft_consensus.cc:3037] T c085eba91aa24acebf42d525e95cd184 P 2bfd1fa177b2471ca35293bbab14f8d0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:04:43.661360 18881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c085eba91aa24acebf42d525e95cd184" candidate_uuid: "f022f0941a3848cb89c09901b649f698" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a3b236510d04c18a3b79f9a59528115"
I20260806 03:04:43.661808 18881 raft_consensus.cc:3037] T c085eba91aa24acebf42d525e95cd184 P 5a3b236510d04c18a3b79f9a59528115 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:04:43.663914 18745 raft_consensus.cc:2454] T c085eba91aa24acebf42d525e95cd184 P 2bfd1fa177b2471ca35293bbab14f8d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f022f0941a3848cb89c09901b649f698 in term 1.
I20260806 03:04:43.664486 18538 leader_election.cc:302] T c085eba91aa24acebf42d525e95cd184 P f022f0941a3848cb89c09901b649f698 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bfd1fa177b2471ca35293bbab14f8d0, f022f0941a3848cb89c09901b649f698; no voters:
I20260806 03:04:43.665073 18956 raft_consensus.cc:2781] T c085eba91aa24acebf42d525e95cd184 P f022f0941a3848cb89c09901b649f698 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:04:43.665424 18881 raft_consensus.cc:2454] T c085eba91aa24acebf42d525e95cd184 P 5a3b236510d04c18a3b79f9a59528115 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f022f0941a3848cb89c09901b649f698 in term 1.
I20260806 03:04:43.665833 18956 raft_consensus.cc:686] T c085eba91aa24acebf42d525e95cd184 P f022f0941a3848cb89c09901b649f698 [term 1 LEADER]: Becoming Leader. State: Replica: f022f0941a3848cb89c09901b649f698, State: Running, Role: LEADER
I20260806 03:04:43.666476 18956 consensus_queue.cc:237] T c085eba91aa24acebf42d525e95cd184 P f022f0941a3848cb89c09901b649f698 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f022f0941a3848cb89c09901b649f698" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 39865 } } peers { permanent_uuid: "2bfd1fa177b2471ca35293bbab14f8d0" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 40771 } } peers { permanent_uuid: "5a3b236510d04c18a3b79f9a59528115" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 40043 } }
I20260806 03:04:43.671828 18458 catalog_manager.cc:5495] T c085eba91aa24acebf42d525e95cd184 P f022f0941a3848cb89c09901b649f698 reported cstate change: term changed from 0 to 1, leader changed from <none> to f022f0941a3848cb89c09901b649f698 (127.31.74.193). New cstate: current_term: 1 leader_uuid: "f022f0941a3848cb89c09901b649f698" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f022f0941a3848cb89c09901b649f698" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 39865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2bfd1fa177b2471ca35293bbab14f8d0" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 40771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a3b236510d04c18a3b79f9a59528115" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 40043 } health_report { overall_health: UNKNOWN } } }
W20260806 03:04:43.690372 18656 tablet.cc:2369] T c085eba91aa24acebf42d525e95cd184 P f022f0941a3848cb89c09901b649f698: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 03:04:43.712926 18944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:43.706673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45366 (local address 127.31.74.193:39865)
0806 03:04:43.706872 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:43.706876 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:43.707040 (+ 164us) server_negotiation.cc:407] Connection header received
0806 03:04:43.707187 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:43.707191 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:43.707304 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:43.707459 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 03:04:43.707952 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:43.708708 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:43.710465 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:43.711016 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:43.711126 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:43.711733 (+ 607us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 03:04:43.712305 (+ 572us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 03:04:43.712445 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:43.712637 (+ 192us) server_negotiation.cc:299] Negotiation successful
0806 03:04:43.712682 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 03:04:43.722683 18881 raft_consensus.cc:1261] T c085eba91aa24acebf42d525e95cd184 P 5a3b236510d04c18a3b79f9a59528115 [term 1 FOLLOWER]: Refusing update from remote peer f022f0941a3848cb89c09901b649f698: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:04:43.722760 18745 raft_consensus.cc:1261] T c085eba91aa24acebf42d525e95cd184 P 2bfd1fa177b2471ca35293bbab14f8d0 [term 1 FOLLOWER]: Refusing update from remote peer f022f0941a3848cb89c09901b649f698: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:04:43.723809 18959 consensus_queue.cc:1035] T c085eba91aa24acebf42d525e95cd184 P f022f0941a3848cb89c09901b649f698 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a3b236510d04c18a3b79f9a59528115" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 40043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:04:43.724211 18956 consensus_queue.cc:1035] T c085eba91aa24acebf42d525e95cd184 P f022f0941a3848cb89c09901b649f698 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2bfd1fa177b2471ca35293bbab14f8d0" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 40771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:04:43.756650 18457 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45148:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 03:04:43.757061 18457 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45148:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 03:04:43.761739 18457 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 32ac7f36a4444784a5e84c7f6304c127: Sending DeleteTablet for 3 replicas of tablet c085eba91aa24acebf42d525e95cd184
I20260806 03:04:43.763114 18861 tablet_service.cc:1514] Processing DeleteTablet for tablet c085eba91aa24acebf42d525e95cd184 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:04:43 UTC) from {username='kudu', principal='kudu/127.31.74.254@KRBTEST.COM'} at 127.0.0.1:52956
I20260806 03:04:43.763192 18587 tablet_service.cc:1514] Processing DeleteTablet for tablet c085eba91aa24acebf42d525e95cd184 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:04:43 UTC) from {username='kudu', principal='kudu/127.31.74.254@KRBTEST.COM'} at 127.0.0.1:45350
I20260806 03:04:43.764159 18724 tablet_service.cc:1514] Processing DeleteTablet for tablet c085eba91aa24acebf42d525e95cd184 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:04:43 UTC) from {username='kudu', principal='kudu/127.31.74.254@KRBTEST.COM'} at 127.0.0.1:47484
I20260806 03:04:43.764902 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 18521
I20260806 03:04:43.787279 18972 tablet_replica.cc:330] T c085eba91aa24acebf42d525e95cd184 P 5a3b236510d04c18a3b79f9a59528115: stopping tablet replica
I20260806 03:04:43.787592 18974 tablet_replica.cc:330] T c085eba91aa24acebf42d525e95cd184 P 2bfd1fa177b2471ca35293bbab14f8d0: stopping tablet replica
I20260806 03:04:43.787874 18972 raft_consensus.cc:2229] T c085eba91aa24acebf42d525e95cd184 P 5a3b236510d04c18a3b79f9a59528115 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:04:43.788107 18974 raft_consensus.cc:2229] T c085eba91aa24acebf42d525e95cd184 P 2bfd1fa177b2471ca35293bbab14f8d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:04:43.788477 18972 raft_consensus.cc:2258] T c085eba91aa24acebf42d525e95cd184 P 5a3b236510d04c18a3b79f9a59528115 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:04:43.788729 18974 raft_consensus.cc:2258] T c085eba91aa24acebf42d525e95cd184 P 2bfd1fa177b2471ca35293bbab14f8d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:04:43.789758 18973 tablet_replica.cc:330] T c085eba91aa24acebf42d525e95cd184 P f022f0941a3848cb89c09901b649f698: stopping tablet replica
I20260806 03:04:43.791925 18972 ts_tablet_manager.cc:1905] T c085eba91aa24acebf42d525e95cd184 P 5a3b236510d04c18a3b79f9a59528115: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:04:43.791925 18974 ts_tablet_manager.cc:1905] T c085eba91aa24acebf42d525e95cd184 P 2bfd1fa177b2471ca35293bbab14f8d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:04:43.792536 18973 raft_consensus.cc:2229] T c085eba91aa24acebf42d525e95cd184 P f022f0941a3848cb89c09901b649f698 [term 1 LEADER]: Raft consensus shutting down.
I20260806 03:04:43.793468 18973 raft_consensus.cc:2258] T c085eba91aa24acebf42d525e95cd184 P f022f0941a3848cb89c09901b649f698 [term 1 FOLLOWER]: Raft consensus is shut down!
Aug 06 03:04:43 dist-test-slave-dhph krb5kdc[18410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985480, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.74.193@KRBTEST.COM
I20260806 03:04:43.799723 18972 ts_tablet_manager.cc:1918] T c085eba91aa24acebf42d525e95cd184 P 5a3b236510d04c18a3b79f9a59528115: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 03:04:43.799952 18972 log.cc:1196] T c085eba91aa24acebf42d525e95cd184 P 5a3b236510d04c18a3b79f9a59528115: Deleting WAL directory at /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/minicluster-data/ts-2/wal/wals/c085eba91aa24acebf42d525e95cd184
I20260806 03:04:43.801074 18972 ts_tablet_manager.cc:1939] T c085eba91aa24acebf42d525e95cd184 P 5a3b236510d04c18a3b79f9a59528115: Deleting consensus metadata
I20260806 03:04:43.802291 18973 ts_tablet_manager.cc:1905] T c085eba91aa24acebf42d525e95cd184 P f022f0941a3848cb89c09901b649f698: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:04:43.803514 18441 catalog_manager.cc:4841] TS 5a3b236510d04c18a3b79f9a59528115 (127.31.74.195:40043): tablet c085eba91aa24acebf42d525e95cd184 (table test-table [id=83b3f671a7db40c684c12c58ca5c1e6f]) successfully deleted
I20260806 03:04:43.805539 18974 ts_tablet_manager.cc:1918] T c085eba91aa24acebf42d525e95cd184 P 2bfd1fa177b2471ca35293bbab14f8d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 03:04:43.805747 18974 log.cc:1196] T c085eba91aa24acebf42d525e95cd184 P 2bfd1fa177b2471ca35293bbab14f8d0: Deleting WAL directory at /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/minicluster-data/ts-1/wal/wals/c085eba91aa24acebf42d525e95cd184
I20260806 03:04:43.806284 18974 ts_tablet_manager.cc:1939] T c085eba91aa24acebf42d525e95cd184 P 2bfd1fa177b2471ca35293bbab14f8d0: Deleting consensus metadata
I20260806 03:04:43.808552 18944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:43.766568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45372 (local address 127.31.74.193:39865)
0806 03:04:43.766836 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:43.766846 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:43.779376 (+ 12530us) server_negotiation.cc:407] Connection header received
0806 03:04:43.780052 (+ 676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:43.780062 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:43.780186 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:43.780301 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:04:43.782818 (+ 2517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:43.784519 (+ 1701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:43.789673 (+ 5154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:43.790632 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:43.790829 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:43.797882 (+ 7053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:04:43.797901 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:04:43.797929 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:04:43.798009 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:04:43.803547 (+ 5538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:04:43.804914 (+ 1367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:04:43.804926 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:04:43.804931 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:04:43.805136 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:04:43.805642 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:04:43.805652 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:04:43.805656 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:04:43.806172 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:04:43.806518 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:43.807687 (+ 1169us) server_negotiation.cc:299] Negotiation successful
0806 03:04:43.808027 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 03:04:43.808544 18442 catalog_manager.cc:4841] TS 2bfd1fa177b2471ca35293bbab14f8d0 (127.31.74.194:40771): tablet c085eba91aa24acebf42d525e95cd184 (table test-table [id=83b3f671a7db40c684c12c58ca5c1e6f]) successfully deleted
I20260806 03:04:43.810446 18649 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 03:04:43.925232 18973 ts_tablet_manager.cc:1918] T c085eba91aa24acebf42d525e95cd184 P f022f0941a3848cb89c09901b649f698: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 03:04:43.925555 18973 log.cc:1196] T c085eba91aa24acebf42d525e95cd184 P f022f0941a3848cb89c09901b649f698: Deleting WAL directory at /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785985105255475-32043-0/minicluster-data/ts-0/wal/wals/c085eba91aa24acebf42d525e95cd184
I20260806 03:04:43.925984 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 18521
W20260806 03:04:43.950706 18444 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.74.193:39865 (error 108)
I20260806 03:04:43.951130 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 18661
I20260806 03:04:43.952852 18978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:43.952051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:39865 (local address 127.0.0.1:45378)
0806 03:04:43.952581 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:43.952692 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.193:39865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":318,"threads_started":1}
W20260806 03:04:43.953159 18444 catalog_manager.cc:4584] TS f022f0941a3848cb89c09901b649f698 (127.31.74.193:39865): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c085eba91aa24acebf42d525e95cd184: Network error: Client connection negotiation failed: client connection to 127.31.74.193:39865: connect: Connection refused (error 111)
Aug 06 03:04:43 dist-test-slave-dhph krb5kdc[18410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985480, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.74.194@KRBTEST.COM
I20260806 03:04:43.969098 18945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:43.952301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47494 (local address 127.31.74.194:40771)
0806 03:04:43.952507 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:43.952513 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:43.952537 (+ 24us) server_negotiation.cc:407] Connection header received
0806 03:04:43.952638 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:43.952643 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:43.952714 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:43.952837 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:04:43.954423 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:43.955289 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:43.956710 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:43.957282 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:43.957449 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:43.962064 (+ 4615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:04:43.962081 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:04:43.962098 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:04:43.962146 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:04:43.965620 (+ 3474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:04:43.966528 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:04:43.966537 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:04:43.966540 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:04:43.966797 (+ 257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:04:43.967232 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:04:43.967237 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:04:43.967243 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:04:43.967680 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:04:43.968037 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:43.968395 (+ 358us) server_negotiation.cc:299] Negotiation successful
0806 03:04:43.968649 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260806 03:04:43.970046 18787 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 03:04:44.007614 18978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:44.006971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:39865 (local address 127.0.0.1:45382)
0806 03:04:44.007225 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:44.007425 (+ 200us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.193:39865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260806 03:04:44.008934 18978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:44.008503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:39865 (local address 127.0.0.1:45388)
0806 03:04:44.008711 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:44.008786 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.193:39865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260806 03:04:44.069741 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 18661
I20260806 03:04:44.071583 18978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:44.069281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:39865 (local address 127.0.0.1:45404)
0806 03:04:44.069478 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:44.069576 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.193:39865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260806 03:04:44.072945 18978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:44.072355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:39865 (local address 127.0.0.1:45408)
0806 03:04:44.072584 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:44.072712 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.193:39865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260806 03:04:44.090765 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 18798
Aug 06 03:04:44 dist-test-slave-dhph krb5kdc[18410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985480, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.74.195@KRBTEST.COM
I20260806 03:04:44.108990 18947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:44.091705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52972 (local address 127.31.74.195:40043)
0806 03:04:44.091908 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:44.091913 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:44.091932 (+ 19us) server_negotiation.cc:407] Connection header received
0806 03:04:44.092134 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:44.092140 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:44.092226 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:44.092326 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:04:44.093771 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:44.094695 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:44.096196 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:44.096633 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:44.096739 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:44.102026 (+ 5287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:04:44.102044 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:04:44.102062 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:04:44.102107 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:04:44.105394 (+ 3287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:04:44.106313 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:04:44.106321 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:04:44.106325 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:04:44.106513 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:04:44.107246 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:04:44.107250 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:04:44.107252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:04:44.107664 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:04:44.107899 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:44.108328 (+ 429us) server_negotiation.cc:299] Negotiation successful
0806 03:04:44.108523 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 03:04:44.110060 18924 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 03:04:44.153743 18978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:44.153230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:39865 (local address 127.0.0.1:45414)
0806 03:04:44.153471 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:44.153581 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.193:39865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260806 03:04:44.155007 18978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:44.154527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:39865 (local address 127.0.0.1:45418)
0806 03:04:44.154777 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:44.154860 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.193:39865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
I20260806 03:04:44.225373 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 18798
I20260806 03:04:44.247687 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 18424
I20260806 03:04:44.249444 18490 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 03:04:44.371315 18978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:44.370608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:39865 (local address 127.0.0.1:45422)
0806 03:04:44.371013 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:44.371154 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.193:39865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227}
I20260806 03:04:44.371918 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 18424
2026-08-06T03:04:44Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (5502 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-06T03:04:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:04:44Z Disabled control of system clock
I20260806 03:04:44.480715 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785985105255475-32043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785985105255475-32043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785985105255475-32043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:46477
--webserver_interface=127.31.74.254
--webserver_port=0
--builtin_ntp_servers=127.31.74.212:41301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.74.254:46477
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 03:04:44.907202 18985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:04:44.907573 18985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:04:44.907809 18985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:04:44.916010 18985 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:04:44.916173 18985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:04:44.916215 18985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:04:44.916270 18985 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:04:44.916316 18985 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:04:44.927538 18985 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:41301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785985105255475-32043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785985105255475-32043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.74.254:46477
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:46477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.74.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785985105255475-32043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:04:44.928918 18985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:04:44.930780 18985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:04:44.945588 18993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:04:44.945736 18990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:04:44.945291 18991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:04:44.947083 18985 server_base.cc:1034] running on GCE node
I20260806 03:04:44.948061 18985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:04:44.949676 18985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:04:44.951085 18985 hybrid_clock.cc:648] HybridClock initialized: now 1785985484951038 us; error 50 us; skew 500 ppm
I20260806 03:04:44.951515 18985 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 03:04:44.954294 18985 webserver.cc:460] Webserver started at http://127.31.74.254:43669/ using document root <none> and password file <none>
I20260806 03:04:44.955152 18985 fs_manager.cc:359] Metadata directory not provided
I20260806 03:04:44.955298 18985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:04:44.955657 18985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:04:44.958824 18985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785985105255475-32043-0/minicluster-data/master-0/data/instance:
uuid: "2ed3f1efed534f7986c2b433fac83245"
format_stamp: "Formatted at 2026-08-06 03:04:44 on dist-test-slave-dhph"
server_key: "58e0562b0d222aba5844899d1f2993a8"
server_key_iv: "e07dead01771a1002723e57788a9b1bf"
server_key_version: "encryptionkey@0"
I20260806 03:04:44.959704 18985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785985105255475-32043-0/minicluster-data/master-0/wal/instance:
uuid: "2ed3f1efed534f7986c2b433fac83245"
format_stamp: "Formatted at 2026-08-06 03:04:44 on dist-test-slave-dhph"
server_key: "58e0562b0d222aba5844899d1f2993a8"
server_key_iv: "e07dead01771a1002723e57788a9b1bf"
server_key_version: "encryptionkey@0"
I20260806 03:04:44.965626 18985 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260806 03:04:44.969592 18999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:04:44.971405 18985 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 03:04:44.971637 18985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785985105255475-32043-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785985105255475-32043-0/minicluster-data/master-0/wal
uuid: "2ed3f1efed534f7986c2b433fac83245"
format_stamp: "Formatted at 2026-08-06 03:04:44 on dist-test-slave-dhph"
server_key: "58e0562b0d222aba5844899d1f2993a8"
server_key_iv: "e07dead01771a1002723e57788a9b1bf"
server_key_version: "encryptionkey@0"
I20260806 03:04:44.971894 18985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785985105255475-32043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785985105255475-32043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785985105255475-32043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:04:44.987011 18985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:04:44.988021 18985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:04:44.988384 18985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:04:45.017015 18985 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.254:46477
I20260806 03:04:45.017038 19052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.254:46477 every 8 connection(s)
I20260806 03:04:45.019295 18985 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
I20260806 03:04:45.021714 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 18985
I20260806 03:04:45.022135 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785985105255475-32043-0/minicluster-data/master-0/wal/instance
I20260806 03:04:45.023015 32043 external_mini_cluster.cc:1428] Setting key 72ca7c0127080090726ea3b73503b982
I20260806 03:04:45.025704 19053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:04:45.043463 19053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ed3f1efed534f7986c2b433fac83245: Bootstrap starting.
I20260806 03:04:45.044788 19058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:45.025430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35136 (local address 127.31.74.254:46477)
0806 03:04:45.027807 (+ 2377us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:45.027819 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:45.028034 (+ 215us) server_negotiation.cc:407] Connection header received
0806 03:04:45.029964 (+ 1930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:45.029987 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:45.030536 (+ 549us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:45.031512 (+ 976us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:04:45.032375 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:45.035082 (+ 2707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:45.037875 (+ 2793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:45.038771 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:45.038957 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:45.041072 (+ 2115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:04:45.041085 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:04:45.041107 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:04:45.042239 (+ 1132us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:04:45.042340 (+ 101us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:04:45.042369 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:04:45.042477 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:45.043057 (+ 580us) server_negotiation.cc:299] Negotiation successful
0806 03:04:45.043679 (+ 622us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1643,"thread_start_us":1317,"threads_started":1}
I20260806 03:04:45.048278 19053 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ed3f1efed534f7986c2b433fac83245: Neither blocks nor log segments found. Creating new log.
I20260806 03:04:45.049971 19053 log.cc:824] T 00000000000000000000000000000000 P 2ed3f1efed534f7986c2b433fac83245: Log is configured to *not* fsync() on all Append() calls
I20260806 03:04:45.054728 19053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ed3f1efed534f7986c2b433fac83245: No bootstrap required, opened a new log
I20260806 03:04:45.066361 19053 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ed3f1efed534f7986c2b433fac83245 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ed3f1efed534f7986c2b433fac83245" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 46477 } }
I20260806 03:04:45.067528 19053 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ed3f1efed534f7986c2b433fac83245 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:04:45.067674 19053 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ed3f1efed534f7986c2b433fac83245 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ed3f1efed534f7986c2b433fac83245, State: Initialized, Role: FOLLOWER
I20260806 03:04:45.068634 19053 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ed3f1efed534f7986c2b433fac83245 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ed3f1efed534f7986c2b433fac83245" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 46477 } }
I20260806 03:04:45.068954 19053 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ed3f1efed534f7986c2b433fac83245 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:04:45.069077 19053 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ed3f1efed534f7986c2b433fac83245 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:04:45.069224 19053 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ed3f1efed534f7986c2b433fac83245 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:04:45.071544 19053 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ed3f1efed534f7986c2b433fac83245 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ed3f1efed534f7986c2b433fac83245" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 46477 } }
I20260806 03:04:45.072049 19053 leader_election.cc:302] T 00000000000000000000000000000000 P 2ed3f1efed534f7986c2b433fac83245 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ed3f1efed534f7986c2b433fac83245; no voters:
I20260806 03:04:45.072674 19053 leader_election.cc:288] T 00000000000000000000000000000000 P 2ed3f1efed534f7986c2b433fac83245 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:04:45.072880 19060 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ed3f1efed534f7986c2b433fac83245 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:04:45.073836 19060 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ed3f1efed534f7986c2b433fac83245 [term 1 LEADER]: Becoming Leader. State: Replica: 2ed3f1efed534f7986c2b433fac83245, State: Running, Role: LEADER
I20260806 03:04:45.074643 19060 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ed3f1efed534f7986c2b433fac83245 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ed3f1efed534f7986c2b433fac83245" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 46477 } }
I20260806 03:04:45.075104 19053 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ed3f1efed534f7986c2b433fac83245 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:04:45.078383 19061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ed3f1efed534f7986c2b433fac83245 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ed3f1efed534f7986c2b433fac83245" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ed3f1efed534f7986c2b433fac83245" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 46477 } } }
I20260806 03:04:45.078991 19061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ed3f1efed534f7986c2b433fac83245 [sys.catalog]: This master's current role is: LEADER
I20260806 03:04:45.078387 19062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ed3f1efed534f7986c2b433fac83245 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ed3f1efed534f7986c2b433fac83245. Latest consensus state: current_term: 1 leader_uuid: "2ed3f1efed534f7986c2b433fac83245" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ed3f1efed534f7986c2b433fac83245" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 46477 } } }
I20260806 03:04:45.079213 19062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ed3f1efed534f7986c2b433fac83245 [sys.catalog]: This master's current role is: LEADER
I20260806 03:04:45.082017 19068 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:04:45.087867 19068 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:04:45.099860 19068 catalog_manager.cc:1347] Generated new cluster ID: 34f5122156b349b4b5ed8ce673c4ec5f
I20260806 03:04:45.100059 19068 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:04:45.127573 19068 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:04:45.129166 19068 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:04:45.147644 19068 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ed3f1efed534f7986c2b433fac83245: Generated new TSK 0
I20260806 03:04:45.148878 19068 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 03:04:45.154805 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785985105255475-32043-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785985105255475-32043-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785985105255475-32043-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.193:0
--local_ip_for_outbound_sockets=127.31.74.193
--webserver_interface=127.31.74.193
--webserver_port=0
--tserver_master_addrs=127.31.74.254:46477
--builtin_ntp_servers=127.31.74.212:41301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 03:04:45.586043 19080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:04:45.586395 19080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:04:45.586494 19080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:04:45.594754 19080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:04:45.594894 19080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:04:45.595009 19080 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.193
I20260806 03:04:45.604888 19080 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:41301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785985105255475-32043-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785985105255475-32043-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.74.193
--webserver_port=0
--tserver_master_addrs=127.31.74.254:46477
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.193
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785985105255475-32043-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:04:45.606628 19080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:04:45.608882 19080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:04:45.624800 19087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:04:45.625489 19088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:04:45.626988 19090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:04:45.629344 19080 server_base.cc:1034] running on GCE node
I20260806 03:04:45.630082 19080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:04:45.631253 19080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:04:45.632544 19080 hybrid_clock.cc:648] HybridClock initialized: now 1785985485632500 us; error 46 us; skew 500 ppm
I20260806 03:04:45.632948 19080 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 03:04:45.635651 19080 webserver.cc:460] Webserver started at http://127.31.74.193:42571/ using document root <none> and password file <none>
I20260806 03:04:45.636423 19080 fs_manager.cc:359] Metadata directory not provided
I20260806 03:04:45.636571 19080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:04:45.636919 19080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:04:45.640709 19080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785985105255475-32043-0/minicluster-data/ts-0/data/instance:
uuid: "e87e5177f38a4c2e849914bc080afb1f"
format_stamp: "Formatted at 2026-08-06 03:04:45 on dist-test-slave-dhph"
server_key: "7818e78f36fdf5e79aba5c147b0d8015"
server_key_iv: "a28d2970f442284c9faa80a08ca674a6"
server_key_version: "encryptionkey@0"
I20260806 03:04:45.641677 19080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785985105255475-32043-0/minicluster-data/ts-0/wal/instance:
uuid: "e87e5177f38a4c2e849914bc080afb1f"
format_stamp: "Formatted at 2026-08-06 03:04:45 on dist-test-slave-dhph"
server_key: "7818e78f36fdf5e79aba5c147b0d8015"
server_key_iv: "a28d2970f442284c9faa80a08ca674a6"
server_key_version: "encryptionkey@0"
I20260806 03:04:45.647802 19080 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260806 03:04:45.651783 19096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:04:45.653579 19080 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260806 03:04:45.653754 19080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785985105255475-32043-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785985105255475-32043-0/minicluster-data/ts-0/wal
uuid: "e87e5177f38a4c2e849914bc080afb1f"
format_stamp: "Formatted at 2026-08-06 03:04:45 on dist-test-slave-dhph"
server_key: "7818e78f36fdf5e79aba5c147b0d8015"
server_key_iv: "a28d2970f442284c9faa80a08ca674a6"
server_key_version: "encryptionkey@0"
I20260806 03:04:45.653963 19080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785985105255475-32043-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785985105255475-32043-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785985105255475-32043-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:04:45.681516 19080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:04:45.682492 19080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:04:45.682943 19080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:04:45.684461 19080 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:04:45.686064 19080 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:04:45.686154 19080 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:04:45.686295 19080 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:04:45.686419 19080 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:04:45.729620 19080 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.193:43363
I20260806 03:04:45.729722 19213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.193:43363 every 8 connection(s)
I20260806 03:04:45.731132 19080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
I20260806 03:04:45.732555 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 19080
I20260806 03:04:45.732753 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785985105255475-32043-0/minicluster-data/ts-0/wal/instance
I20260806 03:04:45.733361 32043 external_mini_cluster.cc:1428] Setting key 5232cda51cd7dfcdb090763e5127aa3f
I20260806 03:04:45.736622 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785985105255475-32043-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785985105255475-32043-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785985105255475-32043-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.194:0
--local_ip_for_outbound_sockets=127.31.74.194
--webserver_interface=127.31.74.194
--webserver_port=0
--tserver_master_addrs=127.31.74.254:46477
--builtin_ntp_servers=127.31.74.212:41301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 03:04:45.746129 19216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:45.734316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.193:48569 (local address 127.31.74.254:46477)
0806 03:04:45.734882 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:45.734887 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:45.736377 (+ 1490us) server_negotiation.cc:407] Connection header received
0806 03:04:45.738330 (+ 1953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:45.738337 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:45.738437 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:45.738575 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:04:45.739882 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:45.740651 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:45.742324 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:45.742882 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:45.743025 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:45.744669 (+ 1644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:04:45.744674 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:04:45.744678 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:04:45.744699 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:04:45.744745 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:04:45.744759 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:04:45.744861 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:45.745198 (+ 337us) server_negotiation.cc:299] Negotiation successful
0806 03:04:45.745325 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":317,"threads_started":1}
I20260806 03:04:45.747527 19217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:45.735009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:46477 (local address 127.31.74.193:48569)
0806 03:04:45.736201 (+ 1192us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:45.736250 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 03:04:45.738016 (+ 1766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:04:45.738911 (+ 895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:04:45.738920 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:04:45.739027 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:04:45.739663 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:45.739693 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:45.740841 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:45.740844 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:45.742167 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:45.742180 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:45.743141 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:45.743148 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:45.743408 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:45.744270 (+ 862us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 03:04:45.744273 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:04:45.744316 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 03:04:45.744410 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:04:45.744863 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:04:45.744878 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:04:45.744888 (+ 10us) client_negotiation.cc:769] Sending connection context
0806 03:04:45.745204 (+ 316us) client_negotiation.cc:240] Negotiation successful
0806 03:04:45.746082 (+ 878us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"thread_start_us":415,"threads_started":1}
I20260806 03:04:45.749518 19214 heartbeater.cc:347] Connected to a master server at 127.31.74.254:46477
I20260806 03:04:45.749955 19214 heartbeater.cc:464] Registering TS with master...
I20260806 03:04:45.750944 19214 heartbeater.cc:511] Master 127.31.74.254:46477 requested a full tablet report, sending...
I20260806 03:04:45.753531 19016 ts_manager.cc:194] Registered new tserver with Master: e87e5177f38a4c2e849914bc080afb1f (127.31.74.193:43363)
I20260806 03:04:45.755266 19016 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.74.193:48569
W20260806 03:04:46.137342 19218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:04:46.137810 19218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:04:46.138010 19218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:04:46.149050 19218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:04:46.149174 19218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:04:46.149364 19218 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.194
I20260806 03:04:46.162232 19218 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:41301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785985105255475-32043-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785985105255475-32043-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.74.194
--webserver_port=0
--tserver_master_addrs=127.31.74.254:46477
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.194
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785985105255475-32043-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:04:46.163703 19218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:04:46.165629 19218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:04:46.180465 19223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:04:46.180802 19224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:04:46.181289 19218 server_base.cc:1034] running on GCE node
W20260806 03:04:46.181416 19226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:04:46.182143 19218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:04:46.183336 19218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:04:46.184607 19218 hybrid_clock.cc:648] HybridClock initialized: now 1785985486184573 us; error 59 us; skew 500 ppm
I20260806 03:04:46.185034 19218 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 03:04:46.188517 19218 webserver.cc:460] Webserver started at http://127.31.74.194:34529/ using document root <none> and password file <none>
I20260806 03:04:46.189491 19218 fs_manager.cc:359] Metadata directory not provided
I20260806 03:04:46.189666 19218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:04:46.190079 19218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:04:46.193485 19218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785985105255475-32043-0/minicluster-data/ts-1/data/instance:
uuid: "a3304bdb8b89418c90018b653b319439"
format_stamp: "Formatted at 2026-08-06 03:04:46 on dist-test-slave-dhph"
server_key: "64d93ba106bb6ee4e11f0d8d776ecb48"
server_key_iv: "3261fa15cebdd429f8c37b9e9aded6f7"
server_key_version: "encryptionkey@0"
I20260806 03:04:46.194383 19218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785985105255475-32043-0/minicluster-data/ts-1/wal/instance:
uuid: "a3304bdb8b89418c90018b653b319439"
format_stamp: "Formatted at 2026-08-06 03:04:46 on dist-test-slave-dhph"
server_key: "64d93ba106bb6ee4e11f0d8d776ecb48"
server_key_iv: "3261fa15cebdd429f8c37b9e9aded6f7"
server_key_version: "encryptionkey@0"
I20260806 03:04:46.200985 19218 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260806 03:04:46.205286 19232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:04:46.207063 19218 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 03:04:46.207476 19218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785985105255475-32043-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785985105255475-32043-0/minicluster-data/ts-1/wal
uuid: "a3304bdb8b89418c90018b653b319439"
format_stamp: "Formatted at 2026-08-06 03:04:46 on dist-test-slave-dhph"
server_key: "64d93ba106bb6ee4e11f0d8d776ecb48"
server_key_iv: "3261fa15cebdd429f8c37b9e9aded6f7"
server_key_version: "encryptionkey@0"
I20260806 03:04:46.207747 19218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785985105255475-32043-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785985105255475-32043-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785985105255475-32043-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:04:46.229377 19218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:04:46.231040 19218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:04:46.231468 19218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:04:46.233011 19218 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:04:46.234808 19218 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:04:46.234948 19218 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:04:46.235097 19218 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:04:46.235201 19218 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:04:46.278187 19218 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.194:38515
I20260806 03:04:46.278290 19349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.194:38515 every 8 connection(s)
I20260806 03:04:46.279701 19218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
I20260806 03:04:46.280663 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 19218
I20260806 03:04:46.280879 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785985105255475-32043-0/minicluster-data/ts-1/wal/instance
I20260806 03:04:46.281692 32043 external_mini_cluster.cc:1428] Setting key 4ef3118b2c9144cecb3527a75d44e162
I20260806 03:04:46.292829 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785985105255475-32043-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785985105255475-32043-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785985105255475-32043-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.195:0
--local_ip_for_outbound_sockets=127.31.74.195
--webserver_interface=127.31.74.195
--webserver_port=0
--tserver_master_addrs=127.31.74.254:46477
--builtin_ntp_servers=127.31.74.212:41301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 03:04:46.301177 19352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:46.283967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.194:52191 (local address 127.31.74.254:46477)
0806 03:04:46.291103 (+ 7136us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:46.291110 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:46.291141 (+ 31us) server_negotiation.cc:407] Connection header received
0806 03:04:46.291198 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:46.291201 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:46.291271 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:46.291376 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:04:46.293283 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:46.294224 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:46.297379 (+ 3155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:46.297882 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:46.298004 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:46.299957 (+ 1953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:04:46.299962 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:04:46.299966 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:04:46.299988 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:04:46.300040 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:04:46.300052 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:04:46.300148 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:46.300495 (+ 347us) server_negotiation.cc:299] Negotiation successful
0806 03:04:46.300614 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6971,"thread_start_us":6817,"threads_started":1}
I20260806 03:04:46.302722 19353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:46.284961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:46477 (local address 127.31.74.194:52191)
0806 03:04:46.286473 (+ 1512us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:46.286531 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 03:04:46.288510 (+ 1979us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:04:46.291864 (+ 3354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:04:46.291880 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:04:46.292029 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:04:46.293013 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:46.293047 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:46.294421 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:46.294431 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:46.297083 (+ 2652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:46.297105 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:46.298159 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:46.298169 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:46.298532 (+ 363us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:46.299496 (+ 964us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 03:04:46.299502 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:04:46.299565 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 03:04:46.299694 (+ 129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:04:46.300167 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:04:46.300180 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:04:46.300191 (+ 11us) client_negotiation.cc:769] Sending connection context
0806 03:04:46.300400 (+ 209us) client_negotiation.cc:240] Negotiation successful
0806 03:04:46.301048 (+ 648us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":949,"thread_start_us":510,"threads_started":1}
I20260806 03:04:46.304507 19350 heartbeater.cc:347] Connected to a master server at 127.31.74.254:46477
I20260806 03:04:46.304891 19350 heartbeater.cc:464] Registering TS with master...
I20260806 03:04:46.305992 19350 heartbeater.cc:511] Master 127.31.74.254:46477 requested a full tablet report, sending...
I20260806 03:04:46.308182 19016 ts_manager.cc:194] Registered new tserver with Master: a3304bdb8b89418c90018b653b319439 (127.31.74.194:38515)
I20260806 03:04:46.309178 19016 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.74.194:52191
W20260806 03:04:46.684247 19354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:04:46.684679 19354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:04:46.684861 19354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:04:46.693688 19354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:04:46.693873 19354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:04:46.694067 19354 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.195
I20260806 03:04:46.703904 19354 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:41301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785985105255475-32043-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785985105255475-32043-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.74.195
--webserver_port=0
--tserver_master_addrs=127.31.74.254:46477
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.195
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785985105255475-32043-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:04:46.705381 19354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:04:46.707441 19354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:04:46.724375 19359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:04:46.724396 19360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:04:46.724464 19362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:04:46.725805 19354 server_base.cc:1034] running on GCE node
I20260806 03:04:46.727026 19354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:04:46.728389 19354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:04:46.729697 19354 hybrid_clock.cc:648] HybridClock initialized: now 1785985486729630 us; error 60 us; skew 500 ppm
I20260806 03:04:46.730145 19354 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 03:04:46.732936 19354 webserver.cc:460] Webserver started at http://127.31.74.195:34609/ using document root <none> and password file <none>
I20260806 03:04:46.733758 19354 fs_manager.cc:359] Metadata directory not provided
I20260806 03:04:46.733865 19354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:04:46.734236 19354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:04:46.737267 19354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785985105255475-32043-0/minicluster-data/ts-2/data/instance:
uuid: "888333011cea428caf0f2b0f874cce72"
format_stamp: "Formatted at 2026-08-06 03:04:46 on dist-test-slave-dhph"
server_key: "a3fa6f499bff3db91f373bc2267630ac"
server_key_iv: "4e37594d4f4f274cbc0eeaf1b7c439b9"
server_key_version: "encryptionkey@0"
I20260806 03:04:46.738142 19354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785985105255475-32043-0/minicluster-data/ts-2/wal/instance:
uuid: "888333011cea428caf0f2b0f874cce72"
format_stamp: "Formatted at 2026-08-06 03:04:46 on dist-test-slave-dhph"
server_key: "a3fa6f499bff3db91f373bc2267630ac"
server_key_iv: "4e37594d4f4f274cbc0eeaf1b7c439b9"
server_key_version: "encryptionkey@0"
I20260806 03:04:46.744359 19354 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.003s
I20260806 03:04:46.749359 19368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:04:46.751081 19354 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260806 03:04:46.751292 19354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785985105255475-32043-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785985105255475-32043-0/minicluster-data/ts-2/wal
uuid: "888333011cea428caf0f2b0f874cce72"
format_stamp: "Formatted at 2026-08-06 03:04:46 on dist-test-slave-dhph"
server_key: "a3fa6f499bff3db91f373bc2267630ac"
server_key_iv: "4e37594d4f4f274cbc0eeaf1b7c439b9"
server_key_version: "encryptionkey@0"
I20260806 03:04:46.751536 19354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785985105255475-32043-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785985105255475-32043-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785985105255475-32043-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:04:46.760286 19214 heartbeater.cc:503] Master 127.31.74.254:46477 was elected leader, sending a full tablet report...
I20260806 03:04:46.767421 19354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:04:46.768457 19354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:04:46.768790 19354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:04:46.769801 19354 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:04:46.771462 19354 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:04:46.771557 19354 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:04:46.771636 19354 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:04:46.771701 19354 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:04:46.811245 19485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.195:46733 every 8 connection(s)
I20260806 03:04:46.811229 19354 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.195:46733
I20260806 03:04:46.813552 19354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
I20260806 03:04:46.819762 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 19354
I20260806 03:04:46.820024 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785985105255475-32043-0/minicluster-data/ts-2/wal/instance
I20260806 03:04:46.820552 32043 external_mini_cluster.cc:1428] Setting key 89d04563b1d51793351d11e80c5c1a86
I20260806 03:04:46.830431 19489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:46.817280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.195:36031 (local address 127.31.74.254:46477)
0806 03:04:46.817819 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:46.817824 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:46.819325 (+ 1501us) server_negotiation.cc:407] Connection header received
0806 03:04:46.821562 (+ 2237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:46.821568 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:46.821665 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:46.821798 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:04:46.823164 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:46.824171 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:46.825983 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:46.826520 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:46.826708 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:46.828740 (+ 2032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:04:46.828747 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:04:46.828751 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:04:46.828783 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:04:46.828848 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:04:46.828865 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:04:46.829271 (+ 406us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:46.829594 (+ 323us) server_negotiation.cc:299] Negotiation successful
0806 03:04:46.829736 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":297,"threads_started":1}
I20260806 03:04:46.832233 19490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:46.817966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:46477 (local address 127.31.74.195:36031)
0806 03:04:46.819159 (+ 1193us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:46.819197 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0806 03:04:46.821247 (+ 2050us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:04:46.822078 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:04:46.822092 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:04:46.822223 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:04:46.822982 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:46.823003 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:46.824340 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:46.824344 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:46.825819 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:46.825833 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:46.826810 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:46.826818 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:46.827076 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:46.828117 (+ 1041us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 03:04:46.828122 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:04:46.828192 (+ 70us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 03:04:46.828333 (+ 141us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:04:46.829260 (+ 927us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:04:46.829279 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:04:46.829292 (+ 13us) client_negotiation.cc:769] Sending connection context
0806 03:04:46.829566 (+ 274us) client_negotiation.cc:240] Negotiation successful
0806 03:04:46.830373 (+ 807us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":345,"threads_started":1}
I20260806 03:04:46.834309 19486 heartbeater.cc:347] Connected to a master server at 127.31.74.254:46477
I20260806 03:04:46.834725 19486 heartbeater.cc:464] Registering TS with master...
I20260806 03:04:46.835767 19486 heartbeater.cc:511] Master 127.31.74.254:46477 requested a full tablet report, sending...
I20260806 03:04:46.838253 19016 ts_manager.cc:194] Registered new tserver with Master: 888333011cea428caf0f2b0f874cce72 (127.31.74.195:46733)
I20260806 03:04:46.839412 32043 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 03:04:46.839488 19016 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.74.195:36031
I20260806 03:04:46.852310 19489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:46.844290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35152 (local address 127.31.74.254:46477)
0806 03:04:46.844507 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:46.844512 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:46.845164 (+ 652us) server_negotiation.cc:407] Connection header received
0806 03:04:46.845247 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:46.845251 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:46.845328 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:46.845441 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:04:46.846144 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:46.847228 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:46.848964 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:46.849415 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:46.849538 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:46.851326 (+ 1788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:04:46.851333 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:04:46.851338 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:04:46.851368 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:04:46.851413 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:04:46.851430 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:04:46.851592 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:46.851944 (+ 352us) server_negotiation.cc:299] Negotiation successful
0806 03:04:46.852013 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260806 03:04:46.857932 19016 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35152:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 03:04:46.865804 19016 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 03:04:46.919921 19500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:46.881507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.194:38515 (local address 127.0.0.1:38172)
0806 03:04:46.899934 (+ 18427us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:46.900029 (+ 95us) client_negotiation.cc:174] Beginning negotiation
0806 03:04:46.900312 (+ 283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:04:46.908786 (+ 8474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:04:46.908796 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:04:46.908836 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:04:46.909472 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:46.909489 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:46.911196 (+ 1707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:46.911201 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:46.915060 (+ 3859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:46.915075 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:46.918776 (+ 3701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:46.918784 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:46.919053 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:46.919131 (+ 78us) client_negotiation.cc:769] Sending connection context
0806 03:04:46.919310 (+ 179us) client_negotiation.cc:240] Negotiation successful
0806 03:04:46.919538 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17929,"thread_start_us":17958,"threads_started":1}
I20260806 03:04:46.921082 19501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:46.882089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38172 (local address 127.31.74.194:38515)
0806 03:04:46.907556 (+ 25467us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:46.907566 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:46.907604 (+ 38us) server_negotiation.cc:407] Connection header received
0806 03:04:46.907850 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:46.907862 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:46.908203 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:46.908486 (+ 283us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:04:46.909743 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:46.911018 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:46.915305 (+ 4287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:46.917494 (+ 2189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:46.918118 (+ 624us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:46.918768 (+ 650us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:46.920410 (+ 1642us) server_negotiation.cc:299] Negotiation successful
0806 03:04:46.920671 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25294,"thread_start_us":25152,"threads_started":1}
I20260806 03:04:46.923269 19504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:46.883920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:43363 (local address 127.0.0.1:54906)
0806 03:04:46.908855 (+ 24935us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:46.908881 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 03:04:46.909041 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:04:46.910310 (+ 1269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:04:46.910318 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:04:46.910357 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:04:46.912076 (+ 1719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:46.912094 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:46.918010 (+ 5916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:46.918017 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:46.920297 (+ 2280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:46.920315 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:46.922442 (+ 2127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:46.922449 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:46.922635 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:46.922668 (+ 33us) client_negotiation.cc:769] Sending connection context
0806 03:04:46.922745 (+ 77us) client_negotiation.cc:240] Negotiation successful
0806 03:04:46.922907 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24797,"mutex_wait_us":1,"thread_start_us":24097,"threads_started":1}
I20260806 03:04:46.925673 19505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:46.883449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.195:46733 (local address 127.0.0.1:58544)
0806 03:04:46.908497 (+ 25048us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:46.908530 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0806 03:04:46.908670 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:04:46.911714 (+ 3044us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:04:46.911721 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:04:46.911754 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:04:46.912394 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:46.912409 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:46.918010 (+ 5601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:46.918017 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:46.921157 (+ 3140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:46.921173 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:46.924785 (+ 3612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:46.924792 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:46.924979 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:46.925015 (+ 36us) client_negotiation.cc:769] Sending connection context
0806 03:04:46.925149 (+ 134us) client_negotiation.cc:240] Negotiation successful
0806 03:04:46.925304 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24866,"mutex_wait_us":33,"thread_start_us":25052,"threads_started":1}
I20260806 03:04:46.923470 19503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:46.882236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54906 (local address 127.31.74.193:43363)
0806 03:04:46.906770 (+ 24534us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:46.906785 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:46.909035 (+ 2250us) server_negotiation.cc:407] Connection header received
0806 03:04:46.909224 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:46.909234 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:46.909468 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:46.909675 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:04:46.912355 (+ 2680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:46.917787 (+ 5432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:46.920600 (+ 2813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:46.922227 (+ 1627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:46.922403 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:46.922815 (+ 412us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:46.922909 (+ 94us) server_negotiation.cc:299] Negotiation successful
0806 03:04:46.923083 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24211,"thread_start_us":24556,"threads_started":1}
I20260806 03:04:46.926641 19281 tablet_service.cc:1467] Processing CreateTablet for tablet 1f01dd66e72a4d5f850adf152cc1c4d6 (DEFAULT_TABLE table=test-table [id=cfb4571219d74793a697c12985955906]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:04:46.927691 19502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:46.882028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58544 (local address 127.31.74.195:46733)
0806 03:04:46.909034 (+ 27006us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:46.909042 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:46.909087 (+ 45us) server_negotiation.cc:407] Connection header received
0806 03:04:46.909479 (+ 392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:46.909486 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:46.910349 (+ 863us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:46.910872 (+ 523us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:04:46.912647 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:46.917787 (+ 5140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:46.921447 (+ 3660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:46.924530 (+ 3083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:46.924677 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:46.924895 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:46.925875 (+ 980us) server_negotiation.cc:299] Negotiation successful
0806 03:04:46.926091 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26811,"thread_start_us":25786,"threads_started":1}
I20260806 03:04:46.928901 19281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f01dd66e72a4d5f850adf152cc1c4d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:04:46.931749 19419 tablet_service.cc:1467] Processing CreateTablet for tablet 1f01dd66e72a4d5f850adf152cc1c4d6 (DEFAULT_TABLE table=test-table [id=cfb4571219d74793a697c12985955906]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:04:46.933437 19419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f01dd66e72a4d5f850adf152cc1c4d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:04:46.932912 19146 tablet_service.cc:1467] Processing CreateTablet for tablet 1f01dd66e72a4d5f850adf152cc1c4d6 (DEFAULT_TABLE table=test-table [id=cfb4571219d74793a697c12985955906]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:04:46.935698 19146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f01dd66e72a4d5f850adf152cc1c4d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:04:46.949532 19506 tablet_bootstrap.cc:492] T 1f01dd66e72a4d5f850adf152cc1c4d6 P 888333011cea428caf0f2b0f874cce72: Bootstrap starting.
I20260806 03:04:46.953194 19506 tablet_bootstrap.cc:654] T 1f01dd66e72a4d5f850adf152cc1c4d6 P 888333011cea428caf0f2b0f874cce72: Neither blocks nor log segments found. Creating new log.
I20260806 03:04:46.954203 19507 tablet_bootstrap.cc:492] T 1f01dd66e72a4d5f850adf152cc1c4d6 P a3304bdb8b89418c90018b653b319439: Bootstrap starting.
I20260806 03:04:46.954716 19506 log.cc:824] T 1f01dd66e72a4d5f850adf152cc1c4d6 P 888333011cea428caf0f2b0f874cce72: Log is configured to *not* fsync() on all Append() calls
I20260806 03:04:46.956457 19508 tablet_bootstrap.cc:492] T 1f01dd66e72a4d5f850adf152cc1c4d6 P e87e5177f38a4c2e849914bc080afb1f: Bootstrap starting.
I20260806 03:04:46.958871 19507 tablet_bootstrap.cc:654] T 1f01dd66e72a4d5f850adf152cc1c4d6 P a3304bdb8b89418c90018b653b319439: Neither blocks nor log segments found. Creating new log.
I20260806 03:04:46.960602 19507 log.cc:824] T 1f01dd66e72a4d5f850adf152cc1c4d6 P a3304bdb8b89418c90018b653b319439: Log is configured to *not* fsync() on all Append() calls
I20260806 03:04:46.960711 19508 tablet_bootstrap.cc:654] T 1f01dd66e72a4d5f850adf152cc1c4d6 P e87e5177f38a4c2e849914bc080afb1f: Neither blocks nor log segments found. Creating new log.
I20260806 03:04:46.960747 19506 tablet_bootstrap.cc:492] T 1f01dd66e72a4d5f850adf152cc1c4d6 P 888333011cea428caf0f2b0f874cce72: No bootstrap required, opened a new log
I20260806 03:04:46.961076 19506 ts_tablet_manager.cc:1397] T 1f01dd66e72a4d5f850adf152cc1c4d6 P 888333011cea428caf0f2b0f874cce72: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.006s
I20260806 03:04:46.962227 19508 log.cc:824] T 1f01dd66e72a4d5f850adf152cc1c4d6 P e87e5177f38a4c2e849914bc080afb1f: Log is configured to *not* fsync() on all Append() calls
I20260806 03:04:46.964172 19507 tablet_bootstrap.cc:492] T 1f01dd66e72a4d5f850adf152cc1c4d6 P a3304bdb8b89418c90018b653b319439: No bootstrap required, opened a new log
I20260806 03:04:46.964489 19507 ts_tablet_manager.cc:1397] T 1f01dd66e72a4d5f850adf152cc1c4d6 P a3304bdb8b89418c90018b653b319439: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.005s
I20260806 03:04:46.965718 19508 tablet_bootstrap.cc:492] T 1f01dd66e72a4d5f850adf152cc1c4d6 P e87e5177f38a4c2e849914bc080afb1f: No bootstrap required, opened a new log
I20260806 03:04:46.966007 19508 ts_tablet_manager.cc:1397] T 1f01dd66e72a4d5f850adf152cc1c4d6 P e87e5177f38a4c2e849914bc080afb1f: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260806 03:04:46.972743 19507 raft_consensus.cc:348] T 1f01dd66e72a4d5f850adf152cc1c4d6 P a3304bdb8b89418c90018b653b319439 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "888333011cea428caf0f2b0f874cce72" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 46733 } } peers { permanent_uuid: "a3304bdb8b89418c90018b653b319439" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 38515 } } peers { permanent_uuid: "e87e5177f38a4c2e849914bc080afb1f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 43363 } }
I20260806 03:04:46.972923 19506 raft_consensus.cc:348] T 1f01dd66e72a4d5f850adf152cc1c4d6 P 888333011cea428caf0f2b0f874cce72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "888333011cea428caf0f2b0f874cce72" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 46733 } } peers { permanent_uuid: "a3304bdb8b89418c90018b653b319439" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 38515 } } peers { permanent_uuid: "e87e5177f38a4c2e849914bc080afb1f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 43363 } }
I20260806 03:04:46.973369 19507 raft_consensus.cc:374] T 1f01dd66e72a4d5f850adf152cc1c4d6 P a3304bdb8b89418c90018b653b319439 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:04:46.973462 19506 raft_consensus.cc:374] T 1f01dd66e72a4d5f850adf152cc1c4d6 P 888333011cea428caf0f2b0f874cce72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:04:46.973515 19507 raft_consensus.cc:729] T 1f01dd66e72a4d5f850adf152cc1c4d6 P a3304bdb8b89418c90018b653b319439 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3304bdb8b89418c90018b653b319439, State: Initialized, Role: FOLLOWER
I20260806 03:04:46.973599 19506 raft_consensus.cc:729] T 1f01dd66e72a4d5f850adf152cc1c4d6 P 888333011cea428caf0f2b0f874cce72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 888333011cea428caf0f2b0f874cce72, State: Initialized, Role: FOLLOWER
I20260806 03:04:46.974330 19506 consensus_queue.cc:260] T 1f01dd66e72a4d5f850adf152cc1c4d6 P 888333011cea428caf0f2b0f874cce72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "888333011cea428caf0f2b0f874cce72" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 46733 } } peers { permanent_uuid: "a3304bdb8b89418c90018b653b319439" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 38515 } } peers { permanent_uuid: "e87e5177f38a4c2e849914bc080afb1f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 43363 } }
I20260806 03:04:46.974349 19507 consensus_queue.cc:260] T 1f01dd66e72a4d5f850adf152cc1c4d6 P a3304bdb8b89418c90018b653b319439 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "888333011cea428caf0f2b0f874cce72" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 46733 } } peers { permanent_uuid: "a3304bdb8b89418c90018b653b319439" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 38515 } } peers { permanent_uuid: "e87e5177f38a4c2e849914bc080afb1f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 43363 } }
I20260806 03:04:46.976672 19486 heartbeater.cc:503] Master 127.31.74.254:46477 was elected leader, sending a full tablet report...
I20260806 03:04:46.977545 19350 heartbeater.cc:503] Master 127.31.74.254:46477 was elected leader, sending a full tablet report...
I20260806 03:04:46.978042 19506 ts_tablet_manager.cc:1428] T 1f01dd66e72a4d5f850adf152cc1c4d6 P 888333011cea428caf0f2b0f874cce72: Time spent starting tablet: real 0.017s user 0.014s sys 0.001s
I20260806 03:04:46.979286 19507 ts_tablet_manager.cc:1428] T 1f01dd66e72a4d5f850adf152cc1c4d6 P a3304bdb8b89418c90018b653b319439: Time spent starting tablet: real 0.015s user 0.008s sys 0.006s
I20260806 03:04:46.981094 19508 raft_consensus.cc:348] T 1f01dd66e72a4d5f850adf152cc1c4d6 P e87e5177f38a4c2e849914bc080afb1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "888333011cea428caf0f2b0f874cce72" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 46733 } } peers { permanent_uuid: "a3304bdb8b89418c90018b653b319439" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 38515 } } peers { permanent_uuid: "e87e5177f38a4c2e849914bc080afb1f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 43363 } }
I20260806 03:04:46.981701 19508 raft_consensus.cc:374] T 1f01dd66e72a4d5f850adf152cc1c4d6 P e87e5177f38a4c2e849914bc080afb1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:04:46.981818 19508 raft_consensus.cc:729] T 1f01dd66e72a4d5f850adf152cc1c4d6 P e87e5177f38a4c2e849914bc080afb1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e87e5177f38a4c2e849914bc080afb1f, State: Initialized, Role: FOLLOWER
I20260806 03:04:46.982578 19508 consensus_queue.cc:260] T 1f01dd66e72a4d5f850adf152cc1c4d6 P e87e5177f38a4c2e849914bc080afb1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "888333011cea428caf0f2b0f874cce72" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 46733 } } peers { permanent_uuid: "a3304bdb8b89418c90018b653b319439" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 38515 } } peers { permanent_uuid: "e87e5177f38a4c2e849914bc080afb1f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 43363 } }
I20260806 03:04:46.985096 19508 ts_tablet_manager.cc:1428] T 1f01dd66e72a4d5f850adf152cc1c4d6 P e87e5177f38a4c2e849914bc080afb1f: Time spent starting tablet: real 0.019s user 0.014s sys 0.004s
W20260806 03:04:47.032613 19351 tablet.cc:2369] T 1f01dd66e72a4d5f850adf152cc1c4d6 P a3304bdb8b89418c90018b653b319439: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 03:04:47.066002 19487 tablet.cc:2369] T 1f01dd66e72a4d5f850adf152cc1c4d6 P 888333011cea428caf0f2b0f874cce72: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 03:04:47.089486 19514 raft_consensus.cc:482] T 1f01dd66e72a4d5f850adf152cc1c4d6 P e87e5177f38a4c2e849914bc080afb1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:04:47.089857 19514 raft_consensus.cc:504] T 1f01dd66e72a4d5f850adf152cc1c4d6 P e87e5177f38a4c2e849914bc080afb1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "888333011cea428caf0f2b0f874cce72" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 46733 } } peers { permanent_uuid: "a3304bdb8b89418c90018b653b319439" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 38515 } } peers { permanent_uuid: "e87e5177f38a4c2e849914bc080afb1f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 43363 } }
I20260806 03:04:47.092705 19514 leader_election.cc:288] T 1f01dd66e72a4d5f850adf152cc1c4d6 P e87e5177f38a4c2e849914bc080afb1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 888333011cea428caf0f2b0f874cce72 (127.31.74.195:46733), a3304bdb8b89418c90018b653b319439 (127.31.74.194:38515)
I20260806 03:04:47.104046 19501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:47.093499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.193:45591 (local address 127.31.74.194:38515)
0806 03:04:47.093699 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:47.093707 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:47.094922 (+ 1215us) server_negotiation.cc:407] Connection header received
0806 03:04:47.095057 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:47.095065 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:47.095167 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:47.095281 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:04:47.096155 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:47.097413 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:47.101324 (+ 3911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:47.102806 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:47.103033 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:47.103126 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:47.103563 (+ 437us) server_negotiation.cc:299] Negotiation successful
0806 03:04:47.103742 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260806 03:04:47.104123 19516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:47.093422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.194:38515 (local address 127.31.74.193:45591)
0806 03:04:47.094801 (+ 1379us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:47.094817 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 03:04:47.094937 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:04:47.095343 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:04:47.095349 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:04:47.095398 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:04:47.095863 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:47.095878 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:47.097686 (+ 1808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:47.097695 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:47.100907 (+ 3212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:47.100930 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:47.103103 (+ 2173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:47.103112 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:47.103352 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:47.103382 (+ 30us) client_negotiation.cc:769] Sending connection context
0806 03:04:47.103472 (+ 90us) client_negotiation.cc:240] Negotiation successful
0806 03:04:47.103583 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1248,"mutex_wait_us":256,"thread_start_us":1012,"threads_started":1}
I20260806 03:04:47.104143 19502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:47.093447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.193:45857 (local address 127.31.74.195:46733)
0806 03:04:47.093700 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:47.093707 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:47.094520 (+ 813us) server_negotiation.cc:407] Connection header received
0806 03:04:47.094863 (+ 343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:47.094889 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:47.094990 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:47.095111 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:04:47.096111 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:47.097410 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:47.101285 (+ 3875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:47.102866 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:47.103083 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:47.103155 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:47.103740 (+ 585us) server_negotiation.cc:299] Negotiation successful
0806 03:04:47.103894 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260806 03:04:47.104224 19515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:47.093274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.195:46733 (local address 127.31.74.193:45857)
0806 03:04:47.094363 (+ 1089us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:47.094389 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 03:04:47.094673 (+ 284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:04:47.095165 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:04:47.095169 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:04:47.095198 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:04:47.095755 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:47.095770 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:47.097708 (+ 1938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:47.097712 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:47.100907 (+ 3195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:47.100930 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:47.103311 (+ 2381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:47.103317 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:47.103497 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:47.103534 (+ 37us) client_negotiation.cc:769] Sending connection context
0806 03:04:47.103641 (+ 107us) client_negotiation.cc:240] Negotiation successful
0806 03:04:47.103781 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":888,"mutex_wait_us":178,"thread_start_us":596,"threads_started":1}
I20260806 03:04:47.105415 19439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f01dd66e72a4d5f850adf152cc1c4d6" candidate_uuid: "e87e5177f38a4c2e849914bc080afb1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "888333011cea428caf0f2b0f874cce72" is_pre_election: true
I20260806 03:04:47.105687 19302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f01dd66e72a4d5f850adf152cc1c4d6" candidate_uuid: "e87e5177f38a4c2e849914bc080afb1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3304bdb8b89418c90018b653b319439" is_pre_election: true
I20260806 03:04:47.106144 19439 raft_consensus.cc:2454] T 1f01dd66e72a4d5f850adf152cc1c4d6 P 888333011cea428caf0f2b0f874cce72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e87e5177f38a4c2e849914bc080afb1f in term 0.
I20260806 03:04:47.106298 19302 raft_consensus.cc:2454] T 1f01dd66e72a4d5f850adf152cc1c4d6 P a3304bdb8b89418c90018b653b319439 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e87e5177f38a4c2e849914bc080afb1f in term 0.
I20260806 03:04:47.107517 19098 leader_election.cc:302] T 1f01dd66e72a4d5f850adf152cc1c4d6 P e87e5177f38a4c2e849914bc080afb1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3304bdb8b89418c90018b653b319439, e87e5177f38a4c2e849914bc080afb1f; no voters:
I20260806 03:04:47.108464 19514 raft_consensus.cc:2781] T 1f01dd66e72a4d5f850adf152cc1c4d6 P e87e5177f38a4c2e849914bc080afb1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:04:47.108742 19514 raft_consensus.cc:482] T 1f01dd66e72a4d5f850adf152cc1c4d6 P e87e5177f38a4c2e849914bc080afb1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:04:47.108911 19514 raft_consensus.cc:3037] T 1f01dd66e72a4d5f850adf152cc1c4d6 P e87e5177f38a4c2e849914bc080afb1f [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:04:47.111658 19514 raft_consensus.cc:504] T 1f01dd66e72a4d5f850adf152cc1c4d6 P e87e5177f38a4c2e849914bc080afb1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "888333011cea428caf0f2b0f874cce72" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 46733 } } peers { permanent_uuid: "a3304bdb8b89418c90018b653b319439" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 38515 } } peers { permanent_uuid: "e87e5177f38a4c2e849914bc080afb1f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 43363 } }
I20260806 03:04:47.112593 19514 leader_election.cc:288] T 1f01dd66e72a4d5f850adf152cc1c4d6 P e87e5177f38a4c2e849914bc080afb1f [CANDIDATE]: Term 1 election: Requested vote from peers 888333011cea428caf0f2b0f874cce72 (127.31.74.195:46733), a3304bdb8b89418c90018b653b319439 (127.31.74.194:38515)
I20260806 03:04:47.113255 19439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f01dd66e72a4d5f850adf152cc1c4d6" candidate_uuid: "e87e5177f38a4c2e849914bc080afb1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "888333011cea428caf0f2b0f874cce72"
I20260806 03:04:47.113312 19302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f01dd66e72a4d5f850adf152cc1c4d6" candidate_uuid: "e87e5177f38a4c2e849914bc080afb1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3304bdb8b89418c90018b653b319439"
I20260806 03:04:47.113451 19439 raft_consensus.cc:3037] T 1f01dd66e72a4d5f850adf152cc1c4d6 P 888333011cea428caf0f2b0f874cce72 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:04:47.113629 19302 raft_consensus.cc:3037] T 1f01dd66e72a4d5f850adf152cc1c4d6 P a3304bdb8b89418c90018b653b319439 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:04:47.116303 19302 raft_consensus.cc:2454] T 1f01dd66e72a4d5f850adf152cc1c4d6 P a3304bdb8b89418c90018b653b319439 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e87e5177f38a4c2e849914bc080afb1f in term 1.
I20260806 03:04:47.116410 19439 raft_consensus.cc:2454] T 1f01dd66e72a4d5f850adf152cc1c4d6 P 888333011cea428caf0f2b0f874cce72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e87e5177f38a4c2e849914bc080afb1f in term 1.
I20260806 03:04:47.117473 19098 leader_election.cc:302] T 1f01dd66e72a4d5f850adf152cc1c4d6 P e87e5177f38a4c2e849914bc080afb1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3304bdb8b89418c90018b653b319439, e87e5177f38a4c2e849914bc080afb1f; no voters:
I20260806 03:04:47.118058 19514 raft_consensus.cc:2781] T 1f01dd66e72a4d5f850adf152cc1c4d6 P e87e5177f38a4c2e849914bc080afb1f [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:04:47.118798 19514 raft_consensus.cc:686] T 1f01dd66e72a4d5f850adf152cc1c4d6 P e87e5177f38a4c2e849914bc080afb1f [term 1 LEADER]: Becoming Leader. State: Replica: e87e5177f38a4c2e849914bc080afb1f, State: Running, Role: LEADER
I20260806 03:04:47.119475 19514 consensus_queue.cc:237] T 1f01dd66e72a4d5f850adf152cc1c4d6 P e87e5177f38a4c2e849914bc080afb1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "888333011cea428caf0f2b0f874cce72" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 46733 } } peers { permanent_uuid: "a3304bdb8b89418c90018b653b319439" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 38515 } } peers { permanent_uuid: "e87e5177f38a4c2e849914bc080afb1f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 43363 } }
I20260806 03:04:47.126165 19016 catalog_manager.cc:5495] T 1f01dd66e72a4d5f850adf152cc1c4d6 P e87e5177f38a4c2e849914bc080afb1f reported cstate change: term changed from 0 to 1, leader changed from <none> to e87e5177f38a4c2e849914bc080afb1f (127.31.74.193). New cstate: current_term: 1 leader_uuid: "e87e5177f38a4c2e849914bc080afb1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "888333011cea428caf0f2b0f874cce72" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 46733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3304bdb8b89418c90018b653b319439" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 38515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e87e5177f38a4c2e849914bc080afb1f" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 43363 } health_report { overall_health: HEALTHY } } }
I20260806 03:04:47.149250 19503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:47.143575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54918 (local address 127.31.74.193:43363)
0806 03:04:47.143803 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:47.143809 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:47.143836 (+ 27us) server_negotiation.cc:407] Connection header received
0806 03:04:47.143953 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:47.143957 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:47.144051 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:47.144217 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 03:04:47.144620 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:47.145378 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:47.146892 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:47.147323 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:47.147480 (+ 157us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:47.148099 (+ 619us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 03:04:47.148620 (+ 521us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 03:04:47.148833 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:47.148948 (+ 115us) server_negotiation.cc:299] Negotiation successful
0806 03:04:47.148993 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 03:04:47.161121 19302 raft_consensus.cc:1261] T 1f01dd66e72a4d5f850adf152cc1c4d6 P a3304bdb8b89418c90018b653b319439 [term 1 FOLLOWER]: Refusing update from remote peer e87e5177f38a4c2e849914bc080afb1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:04:47.161111 19439 raft_consensus.cc:1261] T 1f01dd66e72a4d5f850adf152cc1c4d6 P 888333011cea428caf0f2b0f874cce72 [term 1 FOLLOWER]: Refusing update from remote peer e87e5177f38a4c2e849914bc080afb1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:04:47.162245 19514 consensus_queue.cc:1035] T 1f01dd66e72a4d5f850adf152cc1c4d6 P e87e5177f38a4c2e849914bc080afb1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3304bdb8b89418c90018b653b319439" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 38515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:04:47.162662 19517 consensus_queue.cc:1035] T 1f01dd66e72a4d5f850adf152cc1c4d6 P e87e5177f38a4c2e849914bc080afb1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "888333011cea428caf0f2b0f874cce72" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 46733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:04:47.177481 19520 mvcc.cc:205] Tried to move back new op lower bound from 7315396555396546560 to 7315396555247296512. Current Snapshot: MvccSnapshot[applied={T|T < 7315396555396546560}]
I20260806 03:04:47.197474 19016 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35152:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 03:04:47.198115 19016 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35152:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 03:04:47.203359 19016 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2ed3f1efed534f7986c2b433fac83245: Sending DeleteTablet for 3 replicas of tablet 1f01dd66e72a4d5f850adf152cc1c4d6
I20260806 03:04:47.204731 19281 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f01dd66e72a4d5f850adf152cc1c4d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:04:47 UTC) from {username='slave'} at 127.0.0.1:38172
I20260806 03:04:47.204836 19419 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f01dd66e72a4d5f850adf152cc1c4d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:04:47 UTC) from {username='slave'} at 127.0.0.1:58544
I20260806 03:04:47.207792 19146 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f01dd66e72a4d5f850adf152cc1c4d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:04:47 UTC) from {username='slave'} at 127.0.0.1:54906
I20260806 03:04:47.208245 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 19080
I20260806 03:04:47.212549 19532 tablet_replica.cc:330] T 1f01dd66e72a4d5f850adf152cc1c4d6 P 888333011cea428caf0f2b0f874cce72: stopping tablet replica
I20260806 03:04:47.213140 19532 raft_consensus.cc:2229] T 1f01dd66e72a4d5f850adf152cc1c4d6 P 888333011cea428caf0f2b0f874cce72 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:04:47.213836 19532 raft_consensus.cc:2258] T 1f01dd66e72a4d5f850adf152cc1c4d6 P 888333011cea428caf0f2b0f874cce72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:04:47.215754 19531 tablet_replica.cc:330] T 1f01dd66e72a4d5f850adf152cc1c4d6 P a3304bdb8b89418c90018b653b319439: stopping tablet replica
I20260806 03:04:47.216241 19531 raft_consensus.cc:2229] T 1f01dd66e72a4d5f850adf152cc1c4d6 P a3304bdb8b89418c90018b653b319439 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:04:47.216769 19532 ts_tablet_manager.cc:1905] T 1f01dd66e72a4d5f850adf152cc1c4d6 P 888333011cea428caf0f2b0f874cce72: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:04:47.216879 19531 raft_consensus.cc:2258] T 1f01dd66e72a4d5f850adf152cc1c4d6 P a3304bdb8b89418c90018b653b319439 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:04:47.219306 19531 ts_tablet_manager.cc:1905] T 1f01dd66e72a4d5f850adf152cc1c4d6 P a3304bdb8b89418c90018b653b319439: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:04:47.225643 19532 ts_tablet_manager.cc:1918] T 1f01dd66e72a4d5f850adf152cc1c4d6 P 888333011cea428caf0f2b0f874cce72: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 03:04:47.225883 19503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:47.210661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54932 (local address 127.31.74.193:43363)
0806 03:04:47.210875 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:47.210887 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:47.214973 (+ 4086us) server_negotiation.cc:407] Connection header received
0806 03:04:47.215208 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:47.215217 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:47.215379 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:47.215530 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:04:47.216203 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:47.217642 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:47.219860 (+ 2218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:47.221080 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:47.221207 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:47.223302 (+ 2095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:04:47.223319 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:04:47.223346 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:04:47.224479 (+ 1133us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:04:47.224569 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:04:47.224606 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:04:47.224734 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:47.225094 (+ 360us) server_negotiation.cc:299] Negotiation successful
0806 03:04:47.225277 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 03:04:47.226204 19532 log.cc:1196] T 1f01dd66e72a4d5f850adf152cc1c4d6 P 888333011cea428caf0f2b0f874cce72: Deleting WAL directory at /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785985105255475-32043-0/minicluster-data/ts-2/wal/wals/1f01dd66e72a4d5f850adf152cc1c4d6
I20260806 03:04:47.226962 19532 ts_tablet_manager.cc:1939] T 1f01dd66e72a4d5f850adf152cc1c4d6 P 888333011cea428caf0f2b0f874cce72: Deleting consensus metadata
I20260806 03:04:47.227402 19209 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 03:04:47.228298 19531 ts_tablet_manager.cc:1918] T 1f01dd66e72a4d5f850adf152cc1c4d6 P a3304bdb8b89418c90018b653b319439: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 03:04:47.228585 19531 log.cc:1196] T 1f01dd66e72a4d5f850adf152cc1c4d6 P a3304bdb8b89418c90018b653b319439: Deleting WAL directory at /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785985105255475-32043-0/minicluster-data/ts-1/wal/wals/1f01dd66e72a4d5f850adf152cc1c4d6
I20260806 03:04:47.229331 19531 ts_tablet_manager.cc:1939] T 1f01dd66e72a4d5f850adf152cc1c4d6 P a3304bdb8b89418c90018b653b319439: Deleting consensus metadata
I20260806 03:04:47.229637 19000 catalog_manager.cc:4841] TS 888333011cea428caf0f2b0f874cce72 (127.31.74.195:46733): tablet 1f01dd66e72a4d5f850adf152cc1c4d6 (table test-table [id=cfb4571219d74793a697c12985955906]) successfully deleted
I20260806 03:04:47.231629 19533 tablet_replica.cc:330] T 1f01dd66e72a4d5f850adf152cc1c4d6 P e87e5177f38a4c2e849914bc080afb1f: stopping tablet replica
I20260806 03:04:47.232108 19001 catalog_manager.cc:4841] TS a3304bdb8b89418c90018b653b319439 (127.31.74.194:38515): tablet 1f01dd66e72a4d5f850adf152cc1c4d6 (table test-table [id=cfb4571219d74793a697c12985955906]) successfully deleted
I20260806 03:04:47.232282 19533 raft_consensus.cc:2229] T 1f01dd66e72a4d5f850adf152cc1c4d6 P e87e5177f38a4c2e849914bc080afb1f [term 1 LEADER]: Raft consensus shutting down.
I20260806 03:04:47.334564 19533 raft_consensus.cc:2258] T 1f01dd66e72a4d5f850adf152cc1c4d6 P e87e5177f38a4c2e849914bc080afb1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:04:47.334692 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 19080
I20260806 03:04:47.358954 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 19218
W20260806 03:04:47.359017 19002 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.74.193:43363 (error 108)
I20260806 03:04:47.360859 19505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:47.360398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:43363 (local address 127.0.0.1:50474)
0806 03:04:47.360580 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:47.360709 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.193:43363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260806 03:04:47.361111 19002 catalog_manager.cc:4584] TS e87e5177f38a4c2e849914bc080afb1f (127.31.74.193:43363): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1f01dd66e72a4d5f850adf152cc1c4d6: Network error: Client connection negotiation failed: client connection to 127.31.74.193:43363: connect: Connection refused (error 111)
I20260806 03:04:47.367470 19501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:47.360129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42044 (local address 127.31.74.194:38515)
0806 03:04:47.360345 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:47.360358 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:47.360376 (+ 18us) server_negotiation.cc:407] Connection header received
0806 03:04:47.360434 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:47.360438 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:47.360529 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:47.360653 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:04:47.361183 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:47.362123 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:47.363627 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:47.364116 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:47.364278 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:47.365535 (+ 1257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:04:47.365544 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:04:47.365559 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:04:47.366221 (+ 662us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:04:47.366273 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:04:47.366300 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:04:47.366451 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:47.366739 (+ 288us) server_negotiation.cc:299] Negotiation successful
0806 03:04:47.366863 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260806 03:04:47.368734 19345 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 03:04:47.415791 19505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:47.414987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:43363 (local address 127.0.0.1:50484)
0806 03:04:47.415216 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:47.415305 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.193:43363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260806 03:04:47.417590 19505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:47.416933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:43363 (local address 127.0.0.1:50500)
0806 03:04:47.417273 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:47.417407 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.193:43363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260806 03:04:47.475502 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 19218
I20260806 03:04:47.478860 19505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:47.478205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:43363 (local address 127.0.0.1:50510)
0806 03:04:47.478444 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:47.478573 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.193:43363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260806 03:04:47.480427 19505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:47.479905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:43363 (local address 127.0.0.1:50522)
0806 03:04:47.480114 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:47.480230 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.193:43363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260806 03:04:47.498098 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 19354
I20260806 03:04:47.506225 19502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:47.499128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45934 (local address 127.31.74.195:46733)
0806 03:04:47.499407 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:47.499411 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:47.499526 (+ 115us) server_negotiation.cc:407] Connection header received
0806 03:04:47.499650 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:47.499654 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:47.499760 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:47.499863 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:04:47.500324 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:47.501186 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:47.502787 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:47.503195 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:47.503306 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:47.504665 (+ 1359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:04:47.504674 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:04:47.504691 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:04:47.505310 (+ 619us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:04:47.505361 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:04:47.505381 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:04:47.505497 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:47.505825 (+ 328us) server_negotiation.cc:299] Negotiation successful
0806 03:04:47.505945 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164}
I20260806 03:04:47.507177 19481 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 03:04:47.561095 19505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:47.560561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:43363 (local address 127.0.0.1:50526)
0806 03:04:47.560785 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:47.560899 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.193:43363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260806 03:04:47.562139 19505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:47.561772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:43363 (local address 127.0.0.1:50534)
0806 03:04:47.561969 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:47.562035 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.193:43363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260806 03:04:47.617182 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 19354
I20260806 03:04:47.641083 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 18985
I20260806 03:04:47.643059 19048 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 03:04:47.759215 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 18985
2026-08-06T03:04:47Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3383 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 03:04:47 dist-test-slave-dhph krb5kdc[19542](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 03:04:47 dist-test-slave-dhph krb5kdc[19542](info): set up 2 sockets
Aug 06 03:04:47 dist-test-slave-dhph krb5kdc[19542](info): commencing operation
krb5kdc: starting...
W20260806 03:04:49.052685 32043 mini_kdc.cc:121] Time spent starting KDC: real 1.245s user 0.003s sys 0.056s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/krb5kdc/test-user.keytab.
Aug 06 03:04:49 dist-test-slave-dhph krb5kdc[19542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T03:04:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:04:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.74.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab.
Entry for principal kudu/127.31.74.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab.
I20260806 03:04:49.473129 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:46239
--webserver_interface=127.31.74.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab
--principal=kudu/127.31.74.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:42209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.74.254:46239
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:04:49.900740 19555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:04:49.901113 19555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:04:49.901221 19555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:04:49.912942 19555 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:04:49.913095 19555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:04:49.913146 19555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:04:49.913209 19555 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:04:49.913264 19555 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:04:49.927593 19555 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:42209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.74.254:46239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab
--principal=kudu/127.31.74.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:46239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.74.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:04:49.929065 19555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:04:49.930414 19555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:04:49.948911 19561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:04:49.948962 19563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:04:49.948884 19560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:04:49.951117 19555 server_base.cc:1034] running on GCE node
I20260806 03:04:49.952159 19555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:04:49.954102 19555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:04:49.955410 19555 hybrid_clock.cc:648] HybridClock initialized: now 1785985489955346 us; error 52 us; skew 500 ppm
I20260806 03:04:49.955909 19555 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 03:04:49 dist-test-slave-dhph krb5kdc[19542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985489, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:04:49.964949 19555 init.cc:377] Logged in from keytab as kudu/127.31.74.254@KRBTEST.COM (short username kudu)
I20260806 03:04:49.967459 19555 webserver.cc:460] Webserver started at http://127.31.74.254:40295/ using document root <none> and password file <none>
I20260806 03:04:49.968312 19555 fs_manager.cc:359] Metadata directory not provided
I20260806 03:04:49.968487 19555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:04:49.968998 19555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:04:49.973008 19555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/minicluster-data/master-0/data/instance:
uuid: "4012aa0f11244434a01e5cf9cb446b22"
format_stamp: "Formatted at 2026-08-06 03:04:49 on dist-test-slave-dhph"
server_key: "3e799962ac4ad28eb216bb3314b85ec8"
server_key_iv: "b8c97e819dcdfd3943034c320707c909"
server_key_version: "encryptionkey@0"
I20260806 03:04:49.973922 19555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/minicluster-data/master-0/wal/instance:
uuid: "4012aa0f11244434a01e5cf9cb446b22"
format_stamp: "Formatted at 2026-08-06 03:04:49 on dist-test-slave-dhph"
server_key: "3e799962ac4ad28eb216bb3314b85ec8"
server_key_iv: "b8c97e819dcdfd3943034c320707c909"
server_key_version: "encryptionkey@0"
I20260806 03:04:49.980110 19555 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260806 03:04:49.984670 19570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:04:49.986385 19555 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260806 03:04:49.986716 19555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/minicluster-data/master-0/wal
uuid: "4012aa0f11244434a01e5cf9cb446b22"
format_stamp: "Formatted at 2026-08-06 03:04:49 on dist-test-slave-dhph"
server_key: "3e799962ac4ad28eb216bb3314b85ec8"
server_key_iv: "b8c97e819dcdfd3943034c320707c909"
server_key_version: "encryptionkey@0"
I20260806 03:04:49.986994 19555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:04:50.045823 19555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:04:50.051102 19555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:04:50.051462 19555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:04:50.075840 19555 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.254:46239
I20260806 03:04:50.075850 19623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.254:46239 every 8 connection(s)
I20260806 03:04:50.077337 19555 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
I20260806 03:04:50.081278 19624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:04:50.087175 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 19555
I20260806 03:04:50.087404 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/minicluster-data/master-0/wal/instance
I20260806 03:04:50.087960 32043 external_mini_cluster.cc:1428] Setting key 1453b3488660f8a4983c91193e9274e2
I20260806 03:04:50.093933 19624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4012aa0f11244434a01e5cf9cb446b22: Bootstrap starting.
I20260806 03:04:50.098624 19624 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4012aa0f11244434a01e5cf9cb446b22: Neither blocks nor log segments found. Creating new log.
I20260806 03:04:50.100067 19624 log.cc:824] T 00000000000000000000000000000000 P 4012aa0f11244434a01e5cf9cb446b22: Log is configured to *not* fsync() on all Append() calls
I20260806 03:04:50.104327 19624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4012aa0f11244434a01e5cf9cb446b22: No bootstrap required, opened a new log
Aug 06 03:04:50 dist-test-slave-dhph krb5kdc[19542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 03:04:50.115211 19624 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4012aa0f11244434a01e5cf9cb446b22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4012aa0f11244434a01e5cf9cb446b22" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 46239 } }
I20260806 03:04:50.115826 19624 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4012aa0f11244434a01e5cf9cb446b22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:04:50.115973 19624 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4012aa0f11244434a01e5cf9cb446b22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4012aa0f11244434a01e5cf9cb446b22, State: Initialized, Role: FOLLOWER
I20260806 03:04:50.117192 19627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:50.090204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55966 (local address 127.31.74.254:46239)
0806 03:04:50.091300 (+ 1096us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:50.091353 (+ 53us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:50.091417 (+ 64us) server_negotiation.cc:407] Connection header received
0806 03:04:50.093384 (+ 1967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:50.093406 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:50.093807 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:50.094198 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:04:50.096281 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:50.097897 (+ 1616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:50.100130 (+ 2233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:50.101319 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:50.101576 (+ 257us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:50.107726 (+ 6150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:04:50.107737 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:04:50.107755 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:04:50.107812 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:04:50.111821 (+ 4009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:04:50.112771 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:04:50.112778 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:04:50.112785 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:04:50.112990 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:04:50.113407 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:04:50.113410 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:04:50.113412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:04:50.114034 (+ 622us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:04:50.114345 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:50.115114 (+ 769us) server_negotiation.cc:299] Negotiation successful
0806 03:04:50.115637 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":833,"thread_start_us":459,"threads_started":1}
I20260806 03:04:50.116896 19624 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4012aa0f11244434a01e5cf9cb446b22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4012aa0f11244434a01e5cf9cb446b22" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 46239 } }
I20260806 03:04:50.117453 19624 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4012aa0f11244434a01e5cf9cb446b22 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:04:50.117693 19624 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4012aa0f11244434a01e5cf9cb446b22 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:04:50.117993 19624 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4012aa0f11244434a01e5cf9cb446b22 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:04:50.121178 19624 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4012aa0f11244434a01e5cf9cb446b22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4012aa0f11244434a01e5cf9cb446b22" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 46239 } }
I20260806 03:04:50.121809 19624 leader_election.cc:302] T 00000000000000000000000000000000 P 4012aa0f11244434a01e5cf9cb446b22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4012aa0f11244434a01e5cf9cb446b22; no voters:
I20260806 03:04:50.122898 19624 leader_election.cc:288] T 00000000000000000000000000000000 P 4012aa0f11244434a01e5cf9cb446b22 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:04:50.123193 19629 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4012aa0f11244434a01e5cf9cb446b22 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:04:50.125299 19629 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4012aa0f11244434a01e5cf9cb446b22 [term 1 LEADER]: Becoming Leader. State: Replica: 4012aa0f11244434a01e5cf9cb446b22, State: Running, Role: LEADER
I20260806 03:04:50.126972 19624 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4012aa0f11244434a01e5cf9cb446b22 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:04:50.126892 19629 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4012aa0f11244434a01e5cf9cb446b22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4012aa0f11244434a01e5cf9cb446b22" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 46239 } }
I20260806 03:04:50.131762 19630 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4012aa0f11244434a01e5cf9cb446b22 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4012aa0f11244434a01e5cf9cb446b22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4012aa0f11244434a01e5cf9cb446b22" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 46239 } } }
I20260806 03:04:50.131209 19631 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4012aa0f11244434a01e5cf9cb446b22 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4012aa0f11244434a01e5cf9cb446b22. Latest consensus state: current_term: 1 leader_uuid: "4012aa0f11244434a01e5cf9cb446b22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4012aa0f11244434a01e5cf9cb446b22" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 46239 } } }
I20260806 03:04:50.132525 19630 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4012aa0f11244434a01e5cf9cb446b22 [sys.catalog]: This master's current role is: LEADER
I20260806 03:04:50.132584 19631 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4012aa0f11244434a01e5cf9cb446b22 [sys.catalog]: This master's current role is: LEADER
I20260806 03:04:50.136437 19634 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:04:50.142783 19634 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:04:50.153162 19634 catalog_manager.cc:1347] Generated new cluster ID: 5d3ea9ee354444e28518066fcbdbd912
I20260806 03:04:50.153456 19634 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:04:50.175748 19634 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:04:50.177865 19634 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:04:50.192154 19634 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4012aa0f11244434a01e5cf9cb446b22: Generated new TSK 0
I20260806 03:04:50.194182 19634 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.74.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.193.keytab.
Entry for principal kudu/127.31.74.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.193.keytab.
I20260806 03:04:50.299243 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.193:0
--local_ip_for_outbound_sockets=127.31.74.193
--webserver_interface=127.31.74.193
--webserver_port=0
--tserver_master_addrs=127.31.74.254:46239
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.193.keytab
--principal=kudu/127.31.74.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:42209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:04:50.729666 19652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:04:50.729974 19652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:04:50.730136 19652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:04:50.739744 19652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:04:50.739898 19652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:04:50.740103 19652 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.193
I20260806 03:04:50.752101 19652 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:42209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.193.keytab
--principal=kudu/127.31.74.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.74.193
--webserver_port=0
--tserver_master_addrs=127.31.74.254:46239
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.193
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:04:50.753993 19652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:04:50.755692 19652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:04:50.773026 19660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:04:50.773456 19658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:04:50.772995 19657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:04:50.774370 19652 server_base.cc:1034] running on GCE node
I20260806 03:04:50.775604 19652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:04:50.776942 19652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:04:50.778228 19652 hybrid_clock.cc:648] HybridClock initialized: now 1785985490778180 us; error 43 us; skew 500 ppm
I20260806 03:04:50.778703 19652 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 03:04:50 dist-test-slave-dhph krb5kdc[19542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985490, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:04:50.782721 19652 init.cc:377] Logged in from keytab as kudu/127.31.74.193@KRBTEST.COM (short username kudu)
I20260806 03:04:50.784559 19652 webserver.cc:460] Webserver started at http://127.31.74.193:46865/ using document root <none> and password file <none>
I20260806 03:04:50.785326 19652 fs_manager.cc:359] Metadata directory not provided
I20260806 03:04:50.785447 19652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:04:50.785827 19652 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:04:50.789323 19652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/minicluster-data/ts-0/data/instance:
uuid: "d4f8118618f3427a9ccf0801592f68d6"
format_stamp: "Formatted at 2026-08-06 03:04:50 on dist-test-slave-dhph"
server_key: "e70066b7ab123a35ca4ec44a94b05fed"
server_key_iv: "496647249a665450f728c18b6ecad747"
server_key_version: "encryptionkey@0"
I20260806 03:04:50.790194 19652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/minicluster-data/ts-0/wal/instance:
uuid: "d4f8118618f3427a9ccf0801592f68d6"
format_stamp: "Formatted at 2026-08-06 03:04:50 on dist-test-slave-dhph"
server_key: "e70066b7ab123a35ca4ec44a94b05fed"
server_key_iv: "496647249a665450f728c18b6ecad747"
server_key_version: "encryptionkey@0"
I20260806 03:04:50.796900 19652 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.009s sys 0.000s
I20260806 03:04:50.802068 19667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:04:50.804598 19652 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260806 03:04:50.804827 19652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/minicluster-data/ts-0/wal
uuid: "d4f8118618f3427a9ccf0801592f68d6"
format_stamp: "Formatted at 2026-08-06 03:04:50 on dist-test-slave-dhph"
server_key: "e70066b7ab123a35ca4ec44a94b05fed"
server_key_iv: "496647249a665450f728c18b6ecad747"
server_key_version: "encryptionkey@0"
I20260806 03:04:50.805059 19652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:04:50.843138 19652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:04:50.849088 19652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:04:50.849400 19652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:04:50.850570 19652 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:04:50.852270 19652 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:04:50.852356 19652 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:04:50.852459 19652 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:04:50.852509 19652 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:04:50.894040 19652 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.193:43143
I20260806 03:04:50.894115 19782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.193:43143 every 8 connection(s)
I20260806 03:04:50.895792 19652 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
I20260806 03:04:50.897323 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 19652
I20260806 03:04:50.897740 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/minicluster-data/ts-0/wal/instance
I20260806 03:04:50.898490 32043 external_mini_cluster.cc:1428] Setting key cd2a4c9d8138101fe064ee60be9a75c7
Aug 06 03:04:50 dist-test-slave-dhph krb5kdc[19542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985490, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.193@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 03:04:50.926115 19785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:50.900566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.193:56557 (local address 127.31.74.254:46239)
0806 03:04:50.901200 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:50.901207 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:50.902339 (+ 1132us) server_negotiation.cc:407] Connection header received
0806 03:04:50.904497 (+ 2158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:50.904503 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:50.904608 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:50.904749 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:04:50.907051 (+ 2302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:50.908528 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:50.911402 (+ 2874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:50.912814 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:50.913007 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:50.918427 (+ 5420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:04:50.918512 (+ 85us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:04:50.918527 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:04:50.918853 (+ 326us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:04:50.922390 (+ 3537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:04:50.923480 (+ 1090us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:04:50.923490 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:04:50.923494 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:04:50.923637 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:04:50.924141 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:04:50.924145 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:04:50.924147 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:04:50.924450 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:04:50.924620 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:50.925407 (+ 787us) server_negotiation.cc:299] Negotiation successful
0806 03:04:50.925697 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":299,"threads_started":1}
I20260806 03:04:50.927284 19786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:50.901329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:46239 (local address 127.31.74.193:56557)
0806 03:04:50.902164 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:50.902220 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0806 03:04:50.904176 (+ 1956us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:04:50.905189 (+ 1013us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:04:50.905199 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:04:50.905892 (+ 693us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:04:50.906779 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:50.906808 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:50.908790 (+ 1982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:50.908799 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:50.911149 (+ 2350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:50.911175 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:50.913145 (+ 1970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:50.913154 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:50.913527 (+ 373us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:50.914618 (+ 1091us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:04:50.914665 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:04:50.917668 (+ 3003us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:04:50.922677 (+ 5009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:04:50.922686 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:04:50.922702 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:04:50.923305 (+ 603us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:04:50.923763 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:04:50.923767 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:04:50.923771 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:04:50.923991 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:04:50.924591 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:04:50.924600 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:04:50.925004 (+ 404us) client_negotiation.cc:769] Sending connection context
0806 03:04:50.925305 (+ 301us) client_negotiation.cc:240] Negotiation successful
0806 03:04:50.925911 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":369,"threads_started":1}
I20260806 03:04:50.930146 19783 heartbeater.cc:347] Connected to a master server at 127.31.74.254:46239
I20260806 03:04:50.930852 19783 heartbeater.cc:464] Registering TS with master...
I20260806 03:04:50.931834 19783 heartbeater.cc:511] Master 127.31.74.254:46239 requested a full tablet report, sending...
I20260806 03:04:50.934697 19588 ts_manager.cc:194] Registered new tserver with Master: d4f8118618f3427a9ccf0801592f68d6 (127.31.74.193:43143)
I20260806 03:04:50.937400 19588 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.74.193@KRBTEST.COM'} at 127.31.74.193:56557
WARNING: no policy specified for kudu/127.31.74.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.194.keytab.
Entry for principal kudu/127.31.74.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.194.keytab.
I20260806 03:04:50.991977 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.194:0
--local_ip_for_outbound_sockets=127.31.74.194
--webserver_interface=127.31.74.194
--webserver_port=0
--tserver_master_addrs=127.31.74.254:46239
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.194.keytab
--principal=kudu/127.31.74.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:42209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:04:51.413483 19789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:04:51.414347 19789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:04:51.414543 19789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:04:51.423880 19789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:04:51.424067 19789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:04:51.424244 19789 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.194
I20260806 03:04:51.437384 19789 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:42209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.194.keytab
--principal=kudu/127.31.74.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.74.194
--webserver_port=0
--tserver_master_addrs=127.31.74.254:46239
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.194
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:04:51.438982 19789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:04:51.440444 19789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:04:51.457408 19795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:04:51.457832 19789 server_base.cc:1034] running on GCE node
W20260806 03:04:51.457825 19797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:04:51.457966 19794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:04:51.459056 19789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:04:51.460202 19789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:04:51.461458 19789 hybrid_clock.cc:648] HybridClock initialized: now 1785985491461395 us; error 38 us; skew 500 ppm
I20260806 03:04:51.461781 19789 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 03:04:51 dist-test-slave-dhph krb5kdc[19542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985491, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:04:51.465574 19789 init.cc:377] Logged in from keytab as kudu/127.31.74.194@KRBTEST.COM (short username kudu)
I20260806 03:04:51.467595 19789 webserver.cc:460] Webserver started at http://127.31.74.194:40657/ using document root <none> and password file <none>
I20260806 03:04:51.468349 19789 fs_manager.cc:359] Metadata directory not provided
I20260806 03:04:51.468505 19789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:04:51.468883 19789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:04:51.472215 19789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/minicluster-data/ts-1/data/instance:
uuid: "037c3ac272984919bee2976d41592e77"
format_stamp: "Formatted at 2026-08-06 03:04:51 on dist-test-slave-dhph"
server_key: "54680f94aa03e725bdeedab28e1ccde1"
server_key_iv: "e22b8690b2e2be9b2d5d027abe15b1df"
server_key_version: "encryptionkey@0"
I20260806 03:04:51.473137 19789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/minicluster-data/ts-1/wal/instance:
uuid: "037c3ac272984919bee2976d41592e77"
format_stamp: "Formatted at 2026-08-06 03:04:51 on dist-test-slave-dhph"
server_key: "54680f94aa03e725bdeedab28e1ccde1"
server_key_iv: "e22b8690b2e2be9b2d5d027abe15b1df"
server_key_version: "encryptionkey@0"
I20260806 03:04:51.479250 19789 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260806 03:04:51.483500 19804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:04:51.484967 19789 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 03:04:51.485117 19789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/minicluster-data/ts-1/wal
uuid: "037c3ac272984919bee2976d41592e77"
format_stamp: "Formatted at 2026-08-06 03:04:51 on dist-test-slave-dhph"
server_key: "54680f94aa03e725bdeedab28e1ccde1"
server_key_iv: "e22b8690b2e2be9b2d5d027abe15b1df"
server_key_version: "encryptionkey@0"
I20260806 03:04:51.485361 19789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:04:51.497619 19789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:04:51.502045 19789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:04:51.502353 19789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:04:51.503371 19789 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:04:51.504837 19789 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:04:51.504917 19789 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:04:51.505043 19789 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:04:51.505095 19789 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:04:51.546840 19789 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.194:40389
I20260806 03:04:51.546855 19924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.194:40389 every 8 connection(s)
I20260806 03:04:51.548904 19789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
I20260806 03:04:51.554491 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 19789
I20260806 03:04:51.554749 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/minicluster-data/ts-1/wal/instance
I20260806 03:04:51.555261 32043 external_mini_cluster.cc:1428] Setting key 7e4225be8029cd0f97c4f098a436e7cb
Aug 06 03:04:51 dist-test-slave-dhph krb5kdc[19542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985491, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.194@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 03:04:51.575913 19927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:51.552151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.194:55007 (local address 127.31.74.254:46239)
0806 03:04:51.552698 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:51.552704 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:51.553734 (+ 1030us) server_negotiation.cc:407] Connection header received
0806 03:04:51.557564 (+ 3830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:51.557567 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:51.557688 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:51.557839 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:04:51.560345 (+ 2506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:51.561217 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:51.563778 (+ 2561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:51.564536 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:51.564661 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:51.569368 (+ 4707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:04:51.569374 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:04:51.569378 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:04:51.569420 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:04:51.572528 (+ 3108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:04:51.573392 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:04:51.573396 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:04:51.573399 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:04:51.573529 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:04:51.573977 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:04:51.573987 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:04:51.573989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:04:51.574265 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:04:51.574421 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:51.575201 (+ 780us) server_negotiation.cc:299] Negotiation successful
0806 03:04:51.575454 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":291,"threads_started":1}
I20260806 03:04:51.577140 19928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:51.552670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:46239 (local address 127.31.74.194:55007)
0806 03:04:51.553535 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:51.553587 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 03:04:51.556148 (+ 2561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:04:51.558224 (+ 2076us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:04:51.558239 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:04:51.559173 (+ 934us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:04:51.560110 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:51.560140 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:51.561400 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:51.561408 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:51.563584 (+ 2176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:51.563602 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:51.564803 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:51.564813 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:51.565155 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:51.565978 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:04:51.566015 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:04:51.569035 (+ 3020us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:04:51.572692 (+ 3657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:04:51.572703 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:04:51.572721 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:04:51.573251 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:04:51.573648 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:04:51.573653 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:04:51.573657 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:04:51.573854 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:04:51.574396 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:04:51.574404 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:04:51.574827 (+ 423us) client_negotiation.cc:769] Sending connection context
0806 03:04:51.575104 (+ 277us) client_negotiation.cc:240] Negotiation successful
0806 03:04:51.575679 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":362,"threads_started":1}
I20260806 03:04:51.579020 19925 heartbeater.cc:347] Connected to a master server at 127.31.74.254:46239
I20260806 03:04:51.579444 19925 heartbeater.cc:464] Registering TS with master...
I20260806 03:04:51.580435 19925 heartbeater.cc:511] Master 127.31.74.254:46239 requested a full tablet report, sending...
I20260806 03:04:51.582199 19588 ts_manager.cc:194] Registered new tserver with Master: 037c3ac272984919bee2976d41592e77 (127.31.74.194:40389)
I20260806 03:04:51.583210 19588 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.74.194@KRBTEST.COM'} at 127.31.74.194:55007
WARNING: no policy specified for kudu/127.31.74.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.195.keytab.
Entry for principal kudu/127.31.74.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.195.keytab.
I20260806 03:04:51.652722 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.195:0
--local_ip_for_outbound_sockets=127.31.74.195
--webserver_interface=127.31.74.195
--webserver_port=0
--tserver_master_addrs=127.31.74.254:46239
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.195.keytab
--principal=kudu/127.31.74.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:42209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 03:04:51.941329 19783 heartbeater.cc:503] Master 127.31.74.254:46239 was elected leader, sending a full tablet report...
W20260806 03:04:52.038740 19931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:04:52.039074 19931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:04:52.039247 19931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:04:52.046819 19931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:04:52.046983 19931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:04:52.047197 19931 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.195
I20260806 03:04:52.058552 19931 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:42209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.195.keytab
--principal=kudu/127.31.74.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.74.195
--webserver_port=0
--tserver_master_addrs=127.31.74.254:46239
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.195
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:04:52.060109 19931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:04:52.061591 19931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:04:52.076866 19936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:04:52.076936 19937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:04:52.076880 19939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:04:52.080639 19931 server_base.cc:1034] running on GCE node
I20260806 03:04:52.081462 19931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:04:52.082639 19931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:04:52.083930 19931 hybrid_clock.cc:648] HybridClock initialized: now 1785985492083868 us; error 49 us; skew 500 ppm
I20260806 03:04:52.084350 19931 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 03:04:52 dist-test-slave-dhph krb5kdc[19542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985492, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:04:52.088639 19931 init.cc:377] Logged in from keytab as kudu/127.31.74.195@KRBTEST.COM (short username kudu)
I20260806 03:04:52.090746 19931 webserver.cc:460] Webserver started at http://127.31.74.195:42737/ using document root <none> and password file <none>
I20260806 03:04:52.091635 19931 fs_manager.cc:359] Metadata directory not provided
I20260806 03:04:52.091796 19931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:04:52.092146 19931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:04:52.095824 19931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/minicluster-data/ts-2/data/instance:
uuid: "7f7ce3023da34417902dfa029e8d6981"
format_stamp: "Formatted at 2026-08-06 03:04:52 on dist-test-slave-dhph"
server_key: "f1eff83c9513c45f4bb7abf812531043"
server_key_iv: "9d59e41994eda9ac9d34fbb52da0b70e"
server_key_version: "encryptionkey@0"
I20260806 03:04:52.096722 19931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/minicluster-data/ts-2/wal/instance:
uuid: "7f7ce3023da34417902dfa029e8d6981"
format_stamp: "Formatted at 2026-08-06 03:04:52 on dist-test-slave-dhph"
server_key: "f1eff83c9513c45f4bb7abf812531043"
server_key_iv: "9d59e41994eda9ac9d34fbb52da0b70e"
server_key_version: "encryptionkey@0"
I20260806 03:04:52.102960 19931 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260806 03:04:52.107344 19946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:04:52.108997 19931 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 03:04:52.109177 19931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/minicluster-data/ts-2/wal
uuid: "7f7ce3023da34417902dfa029e8d6981"
format_stamp: "Formatted at 2026-08-06 03:04:52 on dist-test-slave-dhph"
server_key: "f1eff83c9513c45f4bb7abf812531043"
server_key_iv: "9d59e41994eda9ac9d34fbb52da0b70e"
server_key_version: "encryptionkey@0"
I20260806 03:04:52.109411 19931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:04:52.129447 19931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:04:52.134431 19931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:04:52.134792 19931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:04:52.136174 19931 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:04:52.138033 19931 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:04:52.138144 19931 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:04:52.138293 19931 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:04:52.138367 19931 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:04:52.178503 19931 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.195:36801
I20260806 03:04:52.178529 20063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.195:36801 every 8 connection(s)
I20260806 03:04:52.180457 19931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
I20260806 03:04:52.182156 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 19931
I20260806 03:04:52.182413 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/minicluster-data/ts-2/wal/instance
I20260806 03:04:52.183059 32043 external_mini_cluster.cc:1428] Setting key dbc5d216bf39ee75619d81d238793a69
Aug 06 03:04:52 dist-test-slave-dhph krb5kdc[19542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985492, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.195@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 03:04:52.207533 20066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:52.183837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.195:46579 (local address 127.31.74.254:46239)
0806 03:04:52.184330 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:52.184336 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:52.185384 (+ 1048us) server_negotiation.cc:407] Connection header received
0806 03:04:52.187513 (+ 2129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:52.187517 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:52.187592 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:52.187701 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:04:52.190097 (+ 2396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:52.191145 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:52.193488 (+ 2343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:52.193978 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:52.194097 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:52.198749 (+ 4652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:04:52.198757 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:04:52.198761 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:04:52.198807 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:04:52.203459 (+ 4652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:04:52.204469 (+ 1010us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:04:52.204473 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:04:52.204476 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:04:52.204623 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:04:52.205209 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:04:52.205216 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:04:52.205219 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:04:52.205841 (+ 622us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:04:52.206101 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:52.206989 (+ 888us) server_negotiation.cc:299] Negotiation successful
0806 03:04:52.207215 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":204,"threads_started":1}
I20260806 03:04:52.208750 20067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:52.184399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:46239 (local address 127.31.74.195:46579)
0806 03:04:52.185227 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:52.185281 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 03:04:52.187172 (+ 1891us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:04:52.187948 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:04:52.187961 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:04:52.188849 (+ 888us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:04:52.189620 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:52.189642 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:52.191433 (+ 1791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:52.191438 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:52.193299 (+ 1861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:52.193314 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:52.194293 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:52.194300 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:52.194537 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:52.195408 (+ 871us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:04:52.195439 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:04:52.198296 (+ 2857us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:04:52.203690 (+ 5394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:04:52.203700 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:04:52.203716 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:04:52.204322 (+ 606us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:04:52.204755 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:04:52.204759 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:04:52.204761 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:04:52.204993 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:04:52.206071 (+ 1078us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:04:52.206079 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:04:52.206518 (+ 439us) client_negotiation.cc:769] Sending connection context
0806 03:04:52.206892 (+ 374us) client_negotiation.cc:240] Negotiation successful
0806 03:04:52.207442 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":353,"threads_started":1}
I20260806 03:04:52.210860 20064 heartbeater.cc:347] Connected to a master server at 127.31.74.254:46239
I20260806 03:04:52.211227 20064 heartbeater.cc:464] Registering TS with master...
I20260806 03:04:52.212224 20064 heartbeater.cc:511] Master 127.31.74.254:46239 requested a full tablet report, sending...
I20260806 03:04:52.214403 19588 ts_manager.cc:194] Registered new tserver with Master: 7f7ce3023da34417902dfa029e8d6981 (127.31.74.195:36801)
I20260806 03:04:52.215735 19588 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.74.195@KRBTEST.COM'} at 127.31.74.195:46579
I20260806 03:04:52.227035 32043 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 03:04:52.247500 20066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:52.231520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55980 (local address 127.31.74.254:46239)
0806 03:04:52.231853 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:52.231864 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:52.232263 (+ 399us) server_negotiation.cc:407] Connection header received
0806 03:04:52.232426 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:52.232431 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:52.232521 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:52.232613 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:04:52.234233 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:52.235165 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:52.236853 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:52.237339 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:52.237450 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:52.240480 (+ 3030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:04:52.240486 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:04:52.240490 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:04:52.240533 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:04:52.243745 (+ 3212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:04:52.244507 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:04:52.244512 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:04:52.244514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:04:52.244652 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:04:52.245133 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:04:52.245137 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:04:52.245140 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:04:52.245538 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:04:52.245761 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:52.246674 (+ 913us) server_negotiation.cc:299] Negotiation successful
0806 03:04:52.247034 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260806 03:04:52.252866 19588 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55980:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 03:04:52.260789 19588 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 03:04:52.291615 20077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:52.275807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.194:40389 (local address 127.0.0.1:56046)
0806 03:04:52.277342 (+ 1535us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:52.277412 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0806 03:04:52.277649 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:04:52.279030 (+ 1381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:04:52.279044 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:04:52.279092 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:04:52.279949 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:52.279974 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:52.282670 (+ 2696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:52.282680 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:52.286049 (+ 3369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:52.286075 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:52.290007 (+ 3932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:52.290019 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:52.290389 (+ 370us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:52.290479 (+ 90us) client_negotiation.cc:769] Sending connection context
0806 03:04:52.290687 (+ 208us) client_negotiation.cc:240] Negotiation successful
0806 03:04:52.291041 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1187,"mutex_wait_us":1,"thread_start_us":972,"threads_started":1}
I20260806 03:04:52.292234 20078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:52.275930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56046 (local address 127.31.74.194:40389)
0806 03:04:52.276772 (+ 842us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:52.276784 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:52.277910 (+ 1126us) server_negotiation.cc:407] Connection header received
0806 03:04:52.278036 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:52.278044 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:52.278349 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:52.278629 (+ 280us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:04:52.280240 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:52.282399 (+ 2159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:52.286372 (+ 3973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:52.289286 (+ 2914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:52.289472 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:52.289709 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:52.291546 (+ 1837us) server_negotiation.cc:299] Negotiation successful
0806 03:04:52.291792 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":636,"thread_start_us":501,"threads_started":1}
I20260806 03:04:52.296696 19856 tablet_service.cc:1467] Processing CreateTablet for tablet 1cd34e7272b94b72b0d7c160e533ad0f (DEFAULT_TABLE table=test-table [id=65b970f4e7804658a198c0fb8e8b1a3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:04:52.298352 20079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:52.276010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.195:36801 (local address 127.0.0.1:45124)
0806 03:04:52.288569 (+ 12559us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:52.288597 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0806 03:04:52.288879 (+ 282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:04:52.290573 (+ 1694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:04:52.290612 (+ 39us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:04:52.290659 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:04:52.291245 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:52.291259 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:52.293328 (+ 2069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:52.293335 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:52.295366 (+ 2031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:52.295382 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:52.297350 (+ 1968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:52.297358 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:52.297626 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:52.297668 (+ 42us) client_negotiation.cc:769] Sending connection context
0806 03:04:52.297758 (+ 90us) client_negotiation.cc:240] Negotiation successful
0806 03:04:52.297910 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12372,"mutex_wait_us":53,"thread_start_us":13036,"threads_started":1}
I20260806 03:04:52.298856 20080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:52.276590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45124 (local address 127.31.74.195:36801)
0806 03:04:52.278280 (+ 1690us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:52.278291 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:52.288832 (+ 10541us) server_negotiation.cc:407] Connection header received
0806 03:04:52.289094 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:52.289108 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:52.289529 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:52.289976 (+ 447us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:04:52.291497 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:52.293134 (+ 1637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:52.295542 (+ 2408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:52.297118 (+ 1576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:52.297274 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:52.297573 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:52.298162 (+ 589us) server_negotiation.cc:299] Negotiation successful
0806 03:04:52.298379 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1515,"thread_start_us":1335,"threads_started":1}
I20260806 03:04:52.298885 19856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cd34e7272b94b72b0d7c160e533ad0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:04:52.301664 20081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:52.278404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:43143 (local address 127.0.0.1:59084)
0806 03:04:52.290864 (+ 12460us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:52.290887 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 03:04:52.291020 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:04:52.294421 (+ 3401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:04:52.294430 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:04:52.294469 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:04:52.294965 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:52.294979 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:52.296900 (+ 1921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:52.296906 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:52.299020 (+ 2114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:52.299035 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:52.300911 (+ 1876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:52.300917 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:52.301096 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:52.301134 (+ 38us) client_negotiation.cc:769] Sending connection context
0806 03:04:52.301201 (+ 67us) client_negotiation.cc:240] Negotiation successful
0806 03:04:52.301356 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12290,"mutex_wait_us":4,"thread_start_us":12118,"threads_started":1}
I20260806 03:04:52.302666 20082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:52.278856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59084 (local address 127.31.74.193:43143)
0806 03:04:52.293776 (+ 14920us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:52.293785 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:52.293813 (+ 28us) server_negotiation.cc:407] Connection header received
0806 03:04:52.293912 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:52.293920 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:52.294172 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:52.294396 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:04:52.295148 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:52.296751 (+ 1603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:52.299180 (+ 2429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:52.300665 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:52.302019 (+ 1354us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:52.302184 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:52.302248 (+ 64us) server_negotiation.cc:299] Negotiation successful
0806 03:04:52.302366 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14715,"thread_start_us":14521,"threads_started":1}
I20260806 03:04:52.303238 19997 tablet_service.cc:1467] Processing CreateTablet for tablet 1cd34e7272b94b72b0d7c160e533ad0f (DEFAULT_TABLE table=test-table [id=65b970f4e7804658a198c0fb8e8b1a3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:04:52.304982 19997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cd34e7272b94b72b0d7c160e533ad0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:04:52.306075 19716 tablet_service.cc:1467] Processing CreateTablet for tablet 1cd34e7272b94b72b0d7c160e533ad0f (DEFAULT_TABLE table=test-table [id=65b970f4e7804658a198c0fb8e8b1a3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:04:52.307648 19716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cd34e7272b94b72b0d7c160e533ad0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:04:52.316264 20083 tablet_bootstrap.cc:492] T 1cd34e7272b94b72b0d7c160e533ad0f P 037c3ac272984919bee2976d41592e77: Bootstrap starting.
I20260806 03:04:52.320333 20083 tablet_bootstrap.cc:654] T 1cd34e7272b94b72b0d7c160e533ad0f P 037c3ac272984919bee2976d41592e77: Neither blocks nor log segments found. Creating new log.
I20260806 03:04:52.321839 20084 tablet_bootstrap.cc:492] T 1cd34e7272b94b72b0d7c160e533ad0f P 7f7ce3023da34417902dfa029e8d6981: Bootstrap starting.
I20260806 03:04:52.322388 20083 log.cc:824] T 1cd34e7272b94b72b0d7c160e533ad0f P 037c3ac272984919bee2976d41592e77: Log is configured to *not* fsync() on all Append() calls
I20260806 03:04:52.323787 20085 tablet_bootstrap.cc:492] T 1cd34e7272b94b72b0d7c160e533ad0f P d4f8118618f3427a9ccf0801592f68d6: Bootstrap starting.
I20260806 03:04:52.326036 20084 tablet_bootstrap.cc:654] T 1cd34e7272b94b72b0d7c160e533ad0f P 7f7ce3023da34417902dfa029e8d6981: Neither blocks nor log segments found. Creating new log.
I20260806 03:04:52.327801 20084 log.cc:824] T 1cd34e7272b94b72b0d7c160e533ad0f P 7f7ce3023da34417902dfa029e8d6981: Log is configured to *not* fsync() on all Append() calls
I20260806 03:04:52.328260 20085 tablet_bootstrap.cc:654] T 1cd34e7272b94b72b0d7c160e533ad0f P d4f8118618f3427a9ccf0801592f68d6: Neither blocks nor log segments found. Creating new log.
I20260806 03:04:52.330021 20085 log.cc:824] T 1cd34e7272b94b72b0d7c160e533ad0f P d4f8118618f3427a9ccf0801592f68d6: Log is configured to *not* fsync() on all Append() calls
I20260806 03:04:52.330147 20083 tablet_bootstrap.cc:492] T 1cd34e7272b94b72b0d7c160e533ad0f P 037c3ac272984919bee2976d41592e77: No bootstrap required, opened a new log
I20260806 03:04:52.330439 20083 ts_tablet_manager.cc:1397] T 1cd34e7272b94b72b0d7c160e533ad0f P 037c3ac272984919bee2976d41592e77: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.006s
I20260806 03:04:52.333570 20084 tablet_bootstrap.cc:492] T 1cd34e7272b94b72b0d7c160e533ad0f P 7f7ce3023da34417902dfa029e8d6981: No bootstrap required, opened a new log
I20260806 03:04:52.333914 20084 ts_tablet_manager.cc:1397] T 1cd34e7272b94b72b0d7c160e533ad0f P 7f7ce3023da34417902dfa029e8d6981: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
I20260806 03:04:52.333941 20085 tablet_bootstrap.cc:492] T 1cd34e7272b94b72b0d7c160e533ad0f P d4f8118618f3427a9ccf0801592f68d6: No bootstrap required, opened a new log
I20260806 03:04:52.334208 20085 ts_tablet_manager.cc:1397] T 1cd34e7272b94b72b0d7c160e533ad0f P d4f8118618f3427a9ccf0801592f68d6: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.003s
I20260806 03:04:52.342234 20083 raft_consensus.cc:348] T 1cd34e7272b94b72b0d7c160e533ad0f P 037c3ac272984919bee2976d41592e77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "037c3ac272984919bee2976d41592e77" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 40389 } } peers { permanent_uuid: "7f7ce3023da34417902dfa029e8d6981" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 36801 } } peers { permanent_uuid: "d4f8118618f3427a9ccf0801592f68d6" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 43143 } }
I20260806 03:04:52.342869 20083 raft_consensus.cc:374] T 1cd34e7272b94b72b0d7c160e533ad0f P 037c3ac272984919bee2976d41592e77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:04:52.342993 20083 raft_consensus.cc:729] T 1cd34e7272b94b72b0d7c160e533ad0f P 037c3ac272984919bee2976d41592e77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 037c3ac272984919bee2976d41592e77, State: Initialized, Role: FOLLOWER
I20260806 03:04:52.343755 20083 consensus_queue.cc:260] T 1cd34e7272b94b72b0d7c160e533ad0f P 037c3ac272984919bee2976d41592e77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "037c3ac272984919bee2976d41592e77" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 40389 } } peers { permanent_uuid: "7f7ce3023da34417902dfa029e8d6981" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 36801 } } peers { permanent_uuid: "d4f8118618f3427a9ccf0801592f68d6" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 43143 } }
I20260806 03:04:52.343879 20084 raft_consensus.cc:348] T 1cd34e7272b94b72b0d7c160e533ad0f P 7f7ce3023da34417902dfa029e8d6981 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "037c3ac272984919bee2976d41592e77" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 40389 } } peers { permanent_uuid: "7f7ce3023da34417902dfa029e8d6981" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 36801 } } peers { permanent_uuid: "d4f8118618f3427a9ccf0801592f68d6" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 43143 } }
I20260806 03:04:52.344452 20084 raft_consensus.cc:374] T 1cd34e7272b94b72b0d7c160e533ad0f P 7f7ce3023da34417902dfa029e8d6981 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:04:52.344615 20084 raft_consensus.cc:729] T 1cd34e7272b94b72b0d7c160e533ad0f P 7f7ce3023da34417902dfa029e8d6981 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f7ce3023da34417902dfa029e8d6981, State: Initialized, Role: FOLLOWER
I20260806 03:04:52.345427 19925 heartbeater.cc:503] Master 127.31.74.254:46239 was elected leader, sending a full tablet report...
I20260806 03:04:52.346683 20083 ts_tablet_manager.cc:1428] T 1cd34e7272b94b72b0d7c160e533ad0f P 037c3ac272984919bee2976d41592e77: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20260806 03:04:52.346333 20085 raft_consensus.cc:348] T 1cd34e7272b94b72b0d7c160e533ad0f P d4f8118618f3427a9ccf0801592f68d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "037c3ac272984919bee2976d41592e77" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 40389 } } peers { permanent_uuid: "7f7ce3023da34417902dfa029e8d6981" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 36801 } } peers { permanent_uuid: "d4f8118618f3427a9ccf0801592f68d6" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 43143 } }
I20260806 03:04:52.346868 20085 raft_consensus.cc:374] T 1cd34e7272b94b72b0d7c160e533ad0f P d4f8118618f3427a9ccf0801592f68d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:04:52.346995 20085 raft_consensus.cc:729] T 1cd34e7272b94b72b0d7c160e533ad0f P d4f8118618f3427a9ccf0801592f68d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4f8118618f3427a9ccf0801592f68d6, State: Initialized, Role: FOLLOWER
I20260806 03:04:52.346874 20084 consensus_queue.cc:260] T 1cd34e7272b94b72b0d7c160e533ad0f P 7f7ce3023da34417902dfa029e8d6981 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "037c3ac272984919bee2976d41592e77" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 40389 } } peers { permanent_uuid: "7f7ce3023da34417902dfa029e8d6981" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 36801 } } peers { permanent_uuid: "d4f8118618f3427a9ccf0801592f68d6" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 43143 } }
I20260806 03:04:52.347754 20085 consensus_queue.cc:260] T 1cd34e7272b94b72b0d7c160e533ad0f P d4f8118618f3427a9ccf0801592f68d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "037c3ac272984919bee2976d41592e77" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 40389 } } peers { permanent_uuid: "7f7ce3023da34417902dfa029e8d6981" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 36801 } } peers { permanent_uuid: "d4f8118618f3427a9ccf0801592f68d6" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 43143 } }
I20260806 03:04:52.348565 20064 heartbeater.cc:503] Master 127.31.74.254:46239 was elected leader, sending a full tablet report...
I20260806 03:04:52.349455 20084 ts_tablet_manager.cc:1428] T 1cd34e7272b94b72b0d7c160e533ad0f P 7f7ce3023da34417902dfa029e8d6981: Time spent starting tablet: real 0.015s user 0.015s sys 0.001s
I20260806 03:04:52.351408 20085 ts_tablet_manager.cc:1428] T 1cd34e7272b94b72b0d7c160e533ad0f P d4f8118618f3427a9ccf0801592f68d6: Time spent starting tablet: real 0.017s user 0.013s sys 0.006s
W20260806 03:04:52.399353 19784 tablet.cc:2369] T 1cd34e7272b94b72b0d7c160e533ad0f P d4f8118618f3427a9ccf0801592f68d6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 03:04:52.433048 20065 tablet.cc:2369] T 1cd34e7272b94b72b0d7c160e533ad0f P 7f7ce3023da34417902dfa029e8d6981: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 03:04:52.440577 20091 raft_consensus.cc:482] T 1cd34e7272b94b72b0d7c160e533ad0f P d4f8118618f3427a9ccf0801592f68d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:04:52.440990 20091 raft_consensus.cc:504] T 1cd34e7272b94b72b0d7c160e533ad0f P d4f8118618f3427a9ccf0801592f68d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "037c3ac272984919bee2976d41592e77" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 40389 } } peers { permanent_uuid: "7f7ce3023da34417902dfa029e8d6981" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 36801 } } peers { permanent_uuid: "d4f8118618f3427a9ccf0801592f68d6" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 43143 } }
I20260806 03:04:52.443188 20091 leader_election.cc:288] T 1cd34e7272b94b72b0d7c160e533ad0f P d4f8118618f3427a9ccf0801592f68d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 037c3ac272984919bee2976d41592e77 (127.31.74.194:40389), 7f7ce3023da34417902dfa029e8d6981 (127.31.74.195:36801)
I20260806 03:04:52.443737 20090 raft_consensus.cc:482] T 1cd34e7272b94b72b0d7c160e533ad0f P 7f7ce3023da34417902dfa029e8d6981 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:04:52.444185 20090 raft_consensus.cc:504] T 1cd34e7272b94b72b0d7c160e533ad0f P 7f7ce3023da34417902dfa029e8d6981 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "037c3ac272984919bee2976d41592e77" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 40389 } } peers { permanent_uuid: "7f7ce3023da34417902dfa029e8d6981" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 36801 } } peers { permanent_uuid: "d4f8118618f3427a9ccf0801592f68d6" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 43143 } }
I20260806 03:04:52.454682 20080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:52.444079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.193:60423 (local address 127.31.74.195:36801)
0806 03:04:52.444334 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:52.444341 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:52.445138 (+ 797us) server_negotiation.cc:407] Connection header received
0806 03:04:52.445534 (+ 396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:52.445544 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:52.445637 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:52.445711 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:04:52.446817 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:52.448111 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:52.451269 (+ 3158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:52.453413 (+ 2144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:52.453594 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:52.453670 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:52.454228 (+ 558us) server_negotiation.cc:299] Negotiation successful
0806 03:04:52.454390 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260806 03:04:52.455461 20078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:52.444731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.193:47215 (local address 127.31.74.194:40389)
0806 03:04:52.444962 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:52.444968 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:52.446004 (+ 1036us) server_negotiation.cc:407] Connection header received
0806 03:04:52.446475 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:52.446481 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:52.446602 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:52.446685 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:04:52.447538 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:52.448777 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:52.452293 (+ 3516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:52.453886 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:52.454046 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:52.454154 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:52.454884 (+ 730us) server_negotiation.cc:299] Negotiation successful
0806 03:04:52.455095 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260806 03:04:52.455964 20093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:52.443784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.195:36801 (local address 127.31.74.193:60423)
0806 03:04:52.444969 (+ 1185us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:52.444993 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 03:04:52.445304 (+ 311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:04:52.445963 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:04:52.445970 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:04:52.446010 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:04:52.446608 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:52.446627 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:52.448321 (+ 1694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:52.448325 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:52.451056 (+ 2731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:52.451071 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:52.453835 (+ 2764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:52.453842 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:52.454024 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:52.454061 (+ 37us) client_negotiation.cc:769] Sending connection context
0806 03:04:52.455164 (+ 1103us) client_negotiation.cc:240] Negotiation successful
0806 03:04:52.455339 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":997,"mutex_wait_us":3,"thread_start_us":1618,"threads_started":1}
I20260806 03:04:52.448845 20090 leader_election.cc:288] T 1cd34e7272b94b72b0d7c160e533ad0f P 7f7ce3023da34417902dfa029e8d6981 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 037c3ac272984919bee2976d41592e77 (127.31.74.194:40389), d4f8118618f3427a9ccf0801592f68d6 (127.31.74.193:43143)
I20260806 03:04:52.456508 20092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:52.444040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.194:40389 (local address 127.31.74.193:47215)
0806 03:04:52.445860 (+ 1820us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:52.445877 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 03:04:52.446247 (+ 370us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:04:52.446924 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:04:52.446928 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:04:52.446962 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:04:52.447299 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:52.447308 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:52.449003 (+ 1695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:52.449012 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:52.452054 (+ 3042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:52.452071 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:52.454391 (+ 2320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:52.454399 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:52.454661 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:52.454713 (+ 52us) client_negotiation.cc:769] Sending connection context
0806 03:04:52.455812 (+ 1099us) client_negotiation.cc:240] Negotiation successful
0806 03:04:52.456023 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1696,"mutex_wait_us":48,"spinlock_wait_cycles":46848,"thread_start_us":568,"threads_started":1}
I20260806 03:04:52.457398 20018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd34e7272b94b72b0d7c160e533ad0f" candidate_uuid: "d4f8118618f3427a9ccf0801592f68d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f7ce3023da34417902dfa029e8d6981" is_pre_election: true
I20260806 03:04:52.458076 20018 raft_consensus.cc:2454] T 1cd34e7272b94b72b0d7c160e533ad0f P 7f7ce3023da34417902dfa029e8d6981 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4f8118618f3427a9ccf0801592f68d6 in term 0.
I20260806 03:04:52.458488 20067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:52.447974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:43143 (local address 127.31.74.195:59633)
0806 03:04:52.449014 (+ 1040us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:52.449036 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 03:04:52.449156 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:04:52.449685 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:04:52.449691 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:04:52.449723 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:04:52.449954 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:52.449964 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:52.451755 (+ 1791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:52.451761 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:52.454945 (+ 3184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:52.454960 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:52.456885 (+ 1925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:52.456893 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:52.457129 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:52.457184 (+ 55us) client_negotiation.cc:769] Sending connection context
0806 03:04:52.457277 (+ 93us) client_negotiation.cc:240] Negotiation successful
0806 03:04:52.457519 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":909}
I20260806 03:04:52.458465 19876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd34e7272b94b72b0d7c160e533ad0f" candidate_uuid: "d4f8118618f3427a9ccf0801592f68d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "037c3ac272984919bee2976d41592e77" is_pre_election: true
I20260806 03:04:52.458726 20082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:52.448334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.195:59633 (local address 127.31.74.193:43143)
0806 03:04:52.448571 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:52.448580 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:52.449262 (+ 682us) server_negotiation.cc:407] Connection header received
0806 03:04:52.449334 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:52.449338 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:52.449414 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:52.449507 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:04:52.450114 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:52.451414 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:52.455137 (+ 3723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:52.456476 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:52.456655 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:52.456747 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:52.458287 (+ 1540us) server_negotiation.cc:299] Negotiation successful
0806 03:04:52.458445 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 03:04:52.459129 19876 raft_consensus.cc:2454] T 1cd34e7272b94b72b0d7c160e533ad0f P 037c3ac272984919bee2976d41592e77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4f8118618f3427a9ccf0801592f68d6 in term 0.
I20260806 03:04:52.459357 19671 leader_election.cc:302] T 1cd34e7272b94b72b0d7c160e533ad0f P d4f8118618f3427a9ccf0801592f68d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f7ce3023da34417902dfa029e8d6981, d4f8118618f3427a9ccf0801592f68d6; no voters:
I20260806 03:04:52.459951 20091 raft_consensus.cc:2781] T 1cd34e7272b94b72b0d7c160e533ad0f P d4f8118618f3427a9ccf0801592f68d6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:04:52.459791 19737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd34e7272b94b72b0d7c160e533ad0f" candidate_uuid: "7f7ce3023da34417902dfa029e8d6981" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4f8118618f3427a9ccf0801592f68d6" is_pre_election: true
I20260806 03:04:52.460108 20091 raft_consensus.cc:482] T 1cd34e7272b94b72b0d7c160e533ad0f P d4f8118618f3427a9ccf0801592f68d6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:04:52.460222 20091 raft_consensus.cc:3037] T 1cd34e7272b94b72b0d7c160e533ad0f P d4f8118618f3427a9ccf0801592f68d6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:04:52.462891 20091 raft_consensus.cc:504] T 1cd34e7272b94b72b0d7c160e533ad0f P d4f8118618f3427a9ccf0801592f68d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "037c3ac272984919bee2976d41592e77" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 40389 } } peers { permanent_uuid: "7f7ce3023da34417902dfa029e8d6981" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 36801 } } peers { permanent_uuid: "d4f8118618f3427a9ccf0801592f68d6" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 43143 } }
I20260806 03:04:52.463408 19737 raft_consensus.cc:2379] T 1cd34e7272b94b72b0d7c160e533ad0f P d4f8118618f3427a9ccf0801592f68d6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7f7ce3023da34417902dfa029e8d6981 in current term 1: Already voted for candidate d4f8118618f3427a9ccf0801592f68d6 in this term.
I20260806 03:04:52.463816 20091 leader_election.cc:288] T 1cd34e7272b94b72b0d7c160e533ad0f P d4f8118618f3427a9ccf0801592f68d6 [CANDIDATE]: Term 1 election: Requested vote from peers 037c3ac272984919bee2976d41592e77 (127.31.74.194:40389), 7f7ce3023da34417902dfa029e8d6981 (127.31.74.195:36801)
I20260806 03:04:52.464439 20018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd34e7272b94b72b0d7c160e533ad0f" candidate_uuid: "d4f8118618f3427a9ccf0801592f68d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f7ce3023da34417902dfa029e8d6981"
I20260806 03:04:52.464419 19876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd34e7272b94b72b0d7c160e533ad0f" candidate_uuid: "d4f8118618f3427a9ccf0801592f68d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "037c3ac272984919bee2976d41592e77"
I20260806 03:04:52.464679 19876 raft_consensus.cc:3037] T 1cd34e7272b94b72b0d7c160e533ad0f P 037c3ac272984919bee2976d41592e77 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:04:52.464671 20018 raft_consensus.cc:3037] T 1cd34e7272b94b72b0d7c160e533ad0f P 7f7ce3023da34417902dfa029e8d6981 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:04:52.465790 20078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:52.448460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.195:48539 (local address 127.31.74.194:40389)
0806 03:04:52.455751 (+ 7291us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:52.455759 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:52.457197 (+ 1438us) server_negotiation.cc:407] Connection header received
0806 03:04:52.457609 (+ 412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:52.457615 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:52.457704 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:52.457787 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:04:52.459302 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:52.460472 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:52.462292 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:52.464512 (+ 2220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:52.464699 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:52.464780 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:52.465388 (+ 608us) server_negotiation.cc:299] Negotiation successful
0806 03:04:52.465537 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7159,"thread_start_us":9483,"threads_started":1}
I20260806 03:04:52.466473 20096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:52.448026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.194:40389 (local address 127.31.74.195:48539)
0806 03:04:52.457098 (+ 9072us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:52.457118 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 03:04:52.457406 (+ 288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:04:52.458043 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:04:52.458050 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:04:52.458091 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:04:52.459085 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:52.459098 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:52.460679 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:52.460683 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:52.462120 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:52.462134 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:52.464937 (+ 2803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:52.464944 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:52.465166 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:52.465200 (+ 34us) client_negotiation.cc:769] Sending connection context
0806 03:04:52.465978 (+ 778us) client_negotiation.cc:240] Negotiation successful
0806 03:04:52.466147 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8919,"mutex_wait_us":778,"thread_start_us":9183,"threads_started":1}
I20260806 03:04:52.467233 19875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd34e7272b94b72b0d7c160e533ad0f" candidate_uuid: "7f7ce3023da34417902dfa029e8d6981" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "037c3ac272984919bee2976d41592e77" is_pre_election: true
I20260806 03:04:52.468139 19948 leader_election.cc:302] T 1cd34e7272b94b72b0d7c160e533ad0f P 7f7ce3023da34417902dfa029e8d6981 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7f7ce3023da34417902dfa029e8d6981; no voters: 037c3ac272984919bee2976d41592e77, d4f8118618f3427a9ccf0801592f68d6
I20260806 03:04:52.468578 20018 raft_consensus.cc:2454] T 1cd34e7272b94b72b0d7c160e533ad0f P 7f7ce3023da34417902dfa029e8d6981 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4f8118618f3427a9ccf0801592f68d6 in term 1.
I20260806 03:04:52.468590 19876 raft_consensus.cc:2454] T 1cd34e7272b94b72b0d7c160e533ad0f P 037c3ac272984919bee2976d41592e77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4f8118618f3427a9ccf0801592f68d6 in term 1.
I20260806 03:04:52.468860 20090 raft_consensus.cc:2726] T 1cd34e7272b94b72b0d7c160e533ad0f P 7f7ce3023da34417902dfa029e8d6981 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:04:52.469208 19670 leader_election.cc:302] T 1cd34e7272b94b72b0d7c160e533ad0f P d4f8118618f3427a9ccf0801592f68d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 037c3ac272984919bee2976d41592e77, d4f8118618f3427a9ccf0801592f68d6; no voters:
I20260806 03:04:52.469599 20091 raft_consensus.cc:2781] T 1cd34e7272b94b72b0d7c160e533ad0f P d4f8118618f3427a9ccf0801592f68d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:04:52.470182 20091 raft_consensus.cc:686] T 1cd34e7272b94b72b0d7c160e533ad0f P d4f8118618f3427a9ccf0801592f68d6 [term 1 LEADER]: Becoming Leader. State: Replica: d4f8118618f3427a9ccf0801592f68d6, State: Running, Role: LEADER
I20260806 03:04:52.470813 20091 consensus_queue.cc:237] T 1cd34e7272b94b72b0d7c160e533ad0f P d4f8118618f3427a9ccf0801592f68d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "037c3ac272984919bee2976d41592e77" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 40389 } } peers { permanent_uuid: "7f7ce3023da34417902dfa029e8d6981" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 36801 } } peers { permanent_uuid: "d4f8118618f3427a9ccf0801592f68d6" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 43143 } }
I20260806 03:04:52.475447 19586 catalog_manager.cc:5495] T 1cd34e7272b94b72b0d7c160e533ad0f P d4f8118618f3427a9ccf0801592f68d6 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4f8118618f3427a9ccf0801592f68d6 (127.31.74.193). New cstate: current_term: 1 leader_uuid: "d4f8118618f3427a9ccf0801592f68d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "037c3ac272984919bee2976d41592e77" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 40389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f7ce3023da34417902dfa029e8d6981" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 36801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4f8118618f3427a9ccf0801592f68d6" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 43143 } health_report { overall_health: HEALTHY } } }
I20260806 03:04:52.498333 20082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:52.491864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59092 (local address 127.31.74.193:43143)
0806 03:04:52.492049 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:52.492056 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:52.492283 (+ 227us) server_negotiation.cc:407] Connection header received
0806 03:04:52.492425 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:52.492429 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:52.492554 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:52.492744 (+ 190us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 03:04:52.493213 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:52.494400 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:52.496000 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:52.496451 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:52.496560 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:52.497127 (+ 567us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 03:04:52.497678 (+ 551us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 03:04:52.497846 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:52.498008 (+ 162us) server_negotiation.cc:299] Negotiation successful
0806 03:04:52.498080 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 03:04:52.508104 20018 raft_consensus.cc:1261] T 1cd34e7272b94b72b0d7c160e533ad0f P 7f7ce3023da34417902dfa029e8d6981 [term 1 FOLLOWER]: Refusing update from remote peer d4f8118618f3427a9ccf0801592f68d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:04:52.508152 19876 raft_consensus.cc:1261] T 1cd34e7272b94b72b0d7c160e533ad0f P 037c3ac272984919bee2976d41592e77 [term 1 FOLLOWER]: Refusing update from remote peer d4f8118618f3427a9ccf0801592f68d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:04:52.509188 20097 consensus_queue.cc:1035] T 1cd34e7272b94b72b0d7c160e533ad0f P d4f8118618f3427a9ccf0801592f68d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f7ce3023da34417902dfa029e8d6981" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 36801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:04:52.509603 20091 consensus_queue.cc:1035] T 1cd34e7272b94b72b0d7c160e533ad0f P d4f8118618f3427a9ccf0801592f68d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "037c3ac272984919bee2976d41592e77" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 40389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:04:52.521869 20100 mvcc.cc:205] Tried to move back new op lower bound from 7315396577301229568 to 7315396577164660736. Current Snapshot: MvccSnapshot[applied={T|T < 7315396577301229568}]
I20260806 03:04:52.522675 20101 mvcc.cc:205] Tried to move back new op lower bound from 7315396577301229568 to 7315396577164660736. Current Snapshot: MvccSnapshot[applied={T|T < 7315396577301229568}]
I20260806 03:04:52.533536 20102 mvcc.cc:205] Tried to move back new op lower bound from 7315396577301229568 to 7315396577164660736. Current Snapshot: MvccSnapshot[applied={T|T < 7315396577301229568}]
I20260806 03:04:52.540572 19586 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55980:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 03:04:52.540930 19586 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55980:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 03:04:52.544902 19586 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4012aa0f11244434a01e5cf9cb446b22: Sending DeleteTablet for 3 replicas of tablet 1cd34e7272b94b72b0d7c160e533ad0f
I20260806 03:04:52.546993 19997 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cd34e7272b94b72b0d7c160e533ad0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:04:52 UTC) from {username='kudu', principal='kudu/127.31.74.254@KRBTEST.COM'} at 127.0.0.1:45124
I20260806 03:04:52.546995 19856 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cd34e7272b94b72b0d7c160e533ad0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:04:52 UTC) from {username='kudu', principal='kudu/127.31.74.254@KRBTEST.COM'} at 127.0.0.1:56046
I20260806 03:04:52.547832 19716 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cd34e7272b94b72b0d7c160e533ad0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:04:52 UTC) from {username='kudu', principal='kudu/127.31.74.254@KRBTEST.COM'} at 127.0.0.1:59084
I20260806 03:04:52.557117 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 19652
I20260806 03:04:52.569850 20111 tablet_replica.cc:330] T 1cd34e7272b94b72b0d7c160e533ad0f P 7f7ce3023da34417902dfa029e8d6981: stopping tablet replica
I20260806 03:04:52.570608 20111 raft_consensus.cc:2229] T 1cd34e7272b94b72b0d7c160e533ad0f P 7f7ce3023da34417902dfa029e8d6981 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:04:52.571301 20111 raft_consensus.cc:2258] T 1cd34e7272b94b72b0d7c160e533ad0f P 7f7ce3023da34417902dfa029e8d6981 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:04:52.572357 20114 tablet_replica.cc:330] T 1cd34e7272b94b72b0d7c160e533ad0f P d4f8118618f3427a9ccf0801592f68d6: stopping tablet replica
I20260806 03:04:52.572997 20114 raft_consensus.cc:2229] T 1cd34e7272b94b72b0d7c160e533ad0f P d4f8118618f3427a9ccf0801592f68d6 [term 1 LEADER]: Raft consensus shutting down.
I20260806 03:04:52.573926 20114 raft_consensus.cc:2258] T 1cd34e7272b94b72b0d7c160e533ad0f P d4f8118618f3427a9ccf0801592f68d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:04:52.574155 20111 ts_tablet_manager.cc:1905] T 1cd34e7272b94b72b0d7c160e533ad0f P 7f7ce3023da34417902dfa029e8d6981: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:04:52.576272 20114 ts_tablet_manager.cc:1905] T 1cd34e7272b94b72b0d7c160e533ad0f P d4f8118618f3427a9ccf0801592f68d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:04:52.576988 20113 tablet_replica.cc:330] T 1cd34e7272b94b72b0d7c160e533ad0f P 037c3ac272984919bee2976d41592e77: stopping tablet replica
I20260806 03:04:52.577587 20113 raft_consensus.cc:2229] T 1cd34e7272b94b72b0d7c160e533ad0f P 037c3ac272984919bee2976d41592e77 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:04:52.582297 20111 ts_tablet_manager.cc:1918] T 1cd34e7272b94b72b0d7c160e533ad0f P 7f7ce3023da34417902dfa029e8d6981: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 03:04:52.582885 20111 log.cc:1196] T 1cd34e7272b94b72b0d7c160e533ad0f P 7f7ce3023da34417902dfa029e8d6981: Deleting WAL directory at /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/minicluster-data/ts-2/wal/wals/1cd34e7272b94b72b0d7c160e533ad0f
I20260806 03:04:52.583174 20113 raft_consensus.cc:2258] T 1cd34e7272b94b72b0d7c160e533ad0f P 037c3ac272984919bee2976d41592e77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:04:52.583587 20111 ts_tablet_manager.cc:1939] T 1cd34e7272b94b72b0d7c160e533ad0f P 7f7ce3023da34417902dfa029e8d6981: Deleting consensus metadata
I20260806 03:04:52.585528 20113 ts_tablet_manager.cc:1905] T 1cd34e7272b94b72b0d7c160e533ad0f P 037c3ac272984919bee2976d41592e77: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:04:52.585983 19573 catalog_manager.cc:4841] TS 7f7ce3023da34417902dfa029e8d6981 (127.31.74.195:36801): tablet 1cd34e7272b94b72b0d7c160e533ad0f (table test-table [id=65b970f4e7804658a198c0fb8e8b1a3e]) successfully deleted
I20260806 03:04:52.590054 20114 ts_tablet_manager.cc:1918] T 1cd34e7272b94b72b0d7c160e533ad0f P d4f8118618f3427a9ccf0801592f68d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 03:04:52.590341 20114 log.cc:1196] T 1cd34e7272b94b72b0d7c160e533ad0f P d4f8118618f3427a9ccf0801592f68d6: Deleting WAL directory at /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/minicluster-data/ts-0/wal/wals/1cd34e7272b94b72b0d7c160e533ad0f
I20260806 03:04:52.591074 20114 ts_tablet_manager.cc:1939] T 1cd34e7272b94b72b0d7c160e533ad0f P d4f8118618f3427a9ccf0801592f68d6: Deleting consensus metadata
I20260806 03:04:52.593214 19575 catalog_manager.cc:4841] TS d4f8118618f3427a9ccf0801592f68d6 (127.31.74.193:43143): tablet 1cd34e7272b94b72b0d7c160e533ad0f (table test-table [id=65b970f4e7804658a198c0fb8e8b1a3e]) successfully deleted
I20260806 03:04:52.593593 20113 ts_tablet_manager.cc:1918] T 1cd34e7272b94b72b0d7c160e533ad0f P 037c3ac272984919bee2976d41592e77: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 03:04:52.593842 20113 log.cc:1196] T 1cd34e7272b94b72b0d7c160e533ad0f P 037c3ac272984919bee2976d41592e77: Deleting WAL directory at /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785985105255475-32043-0/minicluster-data/ts-1/wal/wals/1cd34e7272b94b72b0d7c160e533ad0f
I20260806 03:04:52.594486 20113 ts_tablet_manager.cc:1939] T 1cd34e7272b94b72b0d7c160e533ad0f P 037c3ac272984919bee2976d41592e77: Deleting consensus metadata
Aug 06 03:04:52 dist-test-slave-dhph krb5kdc[19542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.74.193@KRBTEST.COM
I20260806 03:04:52.596741 19572 catalog_manager.cc:4841] TS 037c3ac272984919bee2976d41592e77 (127.31.74.194:40389): tablet 1cd34e7272b94b72b0d7c160e533ad0f (table test-table [id=65b970f4e7804658a198c0fb8e8b1a3e]) successfully deleted
I20260806 03:04:52.605655 20082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:52.559165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59096 (local address 127.31.74.193:43143)
0806 03:04:52.559452 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:52.559463 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:52.578906 (+ 19443us) server_negotiation.cc:407] Connection header received
0806 03:04:52.579277 (+ 371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:52.579285 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:52.579381 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:52.579466 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:04:52.581384 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:52.582676 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:52.588987 (+ 6311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:52.589730 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:52.589888 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:52.596092 (+ 6204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:04:52.596112 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:04:52.596133 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:04:52.596195 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:04:52.601592 (+ 5397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:04:52.602792 (+ 1200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:04:52.602804 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:04:52.602808 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:04:52.603046 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:04:52.603618 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:04:52.603626 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:04:52.603629 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:04:52.604228 (+ 599us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:04:52.604606 (+ 378us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:52.605002 (+ 396us) server_negotiation.cc:299] Negotiation successful
0806 03:04:52.605272 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260806 03:04:52.606767 19778 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 03:04:52.724640 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 19652
I20260806 03:04:52.751907 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 19789
Aug 06 03:04:52 dist-test-slave-dhph krb5kdc[19542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.74.194@KRBTEST.COM
I20260806 03:04:52.769994 20078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:52.753030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56054 (local address 127.31.74.194:40389)
0806 03:04:52.753212 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:52.753216 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:52.753233 (+ 17us) server_negotiation.cc:407] Connection header received
0806 03:04:52.753292 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:52.753296 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:52.753421 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:52.753514 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:04:52.755224 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:52.756114 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:52.757687 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:52.758346 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:52.758478 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:52.763366 (+ 4888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:04:52.763384 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:04:52.763402 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:04:52.763449 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:04:52.766937 (+ 3488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:04:52.767848 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:04:52.767857 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:04:52.767860 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:04:52.768013 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:04:52.768493 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:04:52.768497 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:04:52.768500 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:04:52.768929 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:04:52.769207 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:52.769496 (+ 289us) server_negotiation.cc:299] Negotiation successful
0806 03:04:52.769729 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 03:04:52.770871 19920 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 03:04:52.868144 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 19789
I20260806 03:04:52.890260 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 19931
Aug 06 03:04:52 dist-test-slave-dhph krb5kdc[19542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.74.195@KRBTEST.COM
I20260806 03:04:52.908792 20080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:52.891270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45132 (local address 127.31.74.195:36801)
0806 03:04:52.891458 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:52.891462 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:52.891514 (+ 52us) server_negotiation.cc:407] Connection header received
0806 03:04:52.891804 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:52.891808 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:52.891890 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:52.891983 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:04:52.893827 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:52.894735 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:52.896157 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:52.896618 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:52.896736 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:52.901647 (+ 4911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:04:52.901665 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:04:52.901683 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:04:52.901730 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:04:52.905115 (+ 3385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:04:52.905942 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:04:52.905950 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:04:52.905953 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:04:52.906113 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:04:52.906898 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:04:52.906902 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:04:52.906905 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:04:52.907523 (+ 618us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:04:52.907885 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:52.908237 (+ 352us) server_negotiation.cc:299] Negotiation successful
0806 03:04:52.908453 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 03:04:52.910105 20059 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 03:04:53.018813 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 19931
I20260806 03:04:53.042164 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 19555
I20260806 03:04:53.043617 19619 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 03:04:53.165848 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 19555
2026-08-06T03:04:53Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (5417 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-06T03:04:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:04:53Z Disabled control of system clock
I20260806 03:04:53.282459 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785985105255475-32043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785985105255475-32043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785985105255475-32043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:36717
--webserver_interface=127.31.74.254
--webserver_port=0
--builtin_ntp_servers=127.31.74.212:34905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.74.254:36717
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 03:04:53.690328 20124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:04:53.690781 20124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:04:53.690946 20124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:04:53.702077 20124 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:04:53.702239 20124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:04:53.702286 20124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:04:53.702397 20124 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:04:53.702486 20124 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:04:53.714713 20124 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:34905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785985105255475-32043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785985105255475-32043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.74.254:36717
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:36717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.74.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785985105255475-32043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:04:53.716601 20124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:04:53.718868 20124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:04:53.733506 20132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:04:53.733625 20129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:04:53.733603 20130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:04:53.734943 20124 server_base.cc:1034] running on GCE node
I20260806 03:04:53.735895 20124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:04:53.737560 20124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:04:53.738888 20124 hybrid_clock.cc:648] HybridClock initialized: now 1785985493738836 us; error 66 us; skew 500 ppm
I20260806 03:04:53.739328 20124 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 03:04:53.742144 20124 webserver.cc:460] Webserver started at http://127.31.74.254:40273/ using document root <none> and password file <none>
I20260806 03:04:53.743032 20124 fs_manager.cc:359] Metadata directory not provided
I20260806 03:04:53.743180 20124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:04:53.743553 20124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:04:53.747219 20124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785985105255475-32043-0/minicluster-data/master-0/data/instance:
uuid: "52cc38329aeb4a229d0672ede00c2da9"
format_stamp: "Formatted at 2026-08-06 03:04:53 on dist-test-slave-dhph"
server_key: "33a55c0faab3b1101a6f47f1c0b9e7db"
server_key_iv: "6e1fa02583cd1ac97e4258cd355f15c0"
server_key_version: "encryptionkey@0"
I20260806 03:04:53.748426 20124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785985105255475-32043-0/minicluster-data/master-0/wal/instance:
uuid: "52cc38329aeb4a229d0672ede00c2da9"
format_stamp: "Formatted at 2026-08-06 03:04:53 on dist-test-slave-dhph"
server_key: "33a55c0faab3b1101a6f47f1c0b9e7db"
server_key_iv: "6e1fa02583cd1ac97e4258cd355f15c0"
server_key_version: "encryptionkey@0"
I20260806 03:04:53.756022 20124 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.005s
I20260806 03:04:53.761451 20138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:04:53.763383 20124 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260806 03:04:53.763628 20124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785985105255475-32043-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785985105255475-32043-0/minicluster-data/master-0/wal
uuid: "52cc38329aeb4a229d0672ede00c2da9"
format_stamp: "Formatted at 2026-08-06 03:04:53 on dist-test-slave-dhph"
server_key: "33a55c0faab3b1101a6f47f1c0b9e7db"
server_key_iv: "6e1fa02583cd1ac97e4258cd355f15c0"
server_key_version: "encryptionkey@0"
I20260806 03:04:53.763937 20124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785985105255475-32043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785985105255475-32043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785985105255475-32043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:04:53.781610 20124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:04:53.782563 20124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:04:53.782941 20124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:04:53.807094 20124 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.254:36717
I20260806 03:04:53.807215 20191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.254:36717 every 8 connection(s)
I20260806 03:04:53.808751 20124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
I20260806 03:04:53.813315 20192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:04:53.814617 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 20124
I20260806 03:04:53.814829 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785985105255475-32043-0/minicluster-data/master-0/wal/instance
I20260806 03:04:53.815438 32043 external_mini_cluster.cc:1428] Setting key 198f762580999b3a30456ddbea93cdf1
I20260806 03:04:53.830060 20192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52cc38329aeb4a229d0672ede00c2da9: Bootstrap starting.
I20260806 03:04:53.833178 20195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:53.817614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41506 (local address 127.31.74.254:36717)
0806 03:04:53.818901 (+ 1287us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:53.818913 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:53.818971 (+ 58us) server_negotiation.cc:407] Connection header received
0806 03:04:53.820829 (+ 1858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:53.820856 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:53.821423 (+ 567us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:53.821808 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:04:53.822685 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:53.824313 (+ 1628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:53.826758 (+ 2445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:53.827635 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:53.827824 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:53.829562 (+ 1738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:04:53.829571 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:04:53.829589 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:04:53.830333 (+ 744us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:04:53.830428 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:04:53.830453 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:04:53.830548 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:53.831063 (+ 515us) server_negotiation.cc:299] Negotiation successful
0806 03:04:53.831656 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":716,"thread_start_us":460,"threads_started":1}
I20260806 03:04:53.833895 20192 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52cc38329aeb4a229d0672ede00c2da9: Neither blocks nor log segments found. Creating new log.
I20260806 03:04:53.835260 20192 log.cc:824] T 00000000000000000000000000000000 P 52cc38329aeb4a229d0672ede00c2da9: Log is configured to *not* fsync() on all Append() calls
I20260806 03:04:53.838687 20192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52cc38329aeb4a229d0672ede00c2da9: No bootstrap required, opened a new log
I20260806 03:04:53.846494 20192 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52cc38329aeb4a229d0672ede00c2da9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52cc38329aeb4a229d0672ede00c2da9" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36717 } }
I20260806 03:04:53.847041 20192 raft_consensus.cc:374] T 00000000000000000000000000000000 P 52cc38329aeb4a229d0672ede00c2da9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:04:53.847174 20192 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52cc38329aeb4a229d0672ede00c2da9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52cc38329aeb4a229d0672ede00c2da9, State: Initialized, Role: FOLLOWER
I20260806 03:04:53.847747 20192 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52cc38329aeb4a229d0672ede00c2da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52cc38329aeb4a229d0672ede00c2da9" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36717 } }
I20260806 03:04:53.848018 20192 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52cc38329aeb4a229d0672ede00c2da9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:04:53.848163 20192 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52cc38329aeb4a229d0672ede00c2da9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:04:53.848331 20192 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52cc38329aeb4a229d0672ede00c2da9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:04:53.850636 20192 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52cc38329aeb4a229d0672ede00c2da9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52cc38329aeb4a229d0672ede00c2da9" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36717 } }
I20260806 03:04:53.851260 20192 leader_election.cc:302] T 00000000000000000000000000000000 P 52cc38329aeb4a229d0672ede00c2da9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52cc38329aeb4a229d0672ede00c2da9; no voters:
I20260806 03:04:53.851976 20192 leader_election.cc:288] T 00000000000000000000000000000000 P 52cc38329aeb4a229d0672ede00c2da9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:04:53.852156 20197 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52cc38329aeb4a229d0672ede00c2da9 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:04:53.853067 20197 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52cc38329aeb4a229d0672ede00c2da9 [term 1 LEADER]: Becoming Leader. State: Replica: 52cc38329aeb4a229d0672ede00c2da9, State: Running, Role: LEADER
I20260806 03:04:53.853730 20197 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52cc38329aeb4a229d0672ede00c2da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52cc38329aeb4a229d0672ede00c2da9" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36717 } }
I20260806 03:04:53.854307 20192 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52cc38329aeb4a229d0672ede00c2da9 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:04:53.857154 20199 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52cc38329aeb4a229d0672ede00c2da9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52cc38329aeb4a229d0672ede00c2da9. Latest consensus state: current_term: 1 leader_uuid: "52cc38329aeb4a229d0672ede00c2da9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52cc38329aeb4a229d0672ede00c2da9" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36717 } } }
I20260806 03:04:53.857949 20199 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52cc38329aeb4a229d0672ede00c2da9 [sys.catalog]: This master's current role is: LEADER
I20260806 03:04:53.858479 20198 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52cc38329aeb4a229d0672ede00c2da9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "52cc38329aeb4a229d0672ede00c2da9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52cc38329aeb4a229d0672ede00c2da9" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36717 } } }
I20260806 03:04:53.859023 20198 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52cc38329aeb4a229d0672ede00c2da9 [sys.catalog]: This master's current role is: LEADER
I20260806 03:04:53.861374 20206 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:04:53.867460 20206 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:04:53.878680 20206 catalog_manager.cc:1347] Generated new cluster ID: 49855fe2c6a048a7bfc828eef43cd63c
I20260806 03:04:53.878912 20206 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:04:53.889348 20206 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:04:53.890813 20206 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:04:53.902487 20206 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 52cc38329aeb4a229d0672ede00c2da9: Generated new TSK 0
I20260806 03:04:53.904160 20206 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 03:04:53.914533 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785985105255475-32043-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785985105255475-32043-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785985105255475-32043-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.193:0
--local_ip_for_outbound_sockets=127.31.74.193
--webserver_interface=127.31.74.193
--webserver_port=0
--tserver_master_addrs=127.31.74.254:36717
--builtin_ntp_servers=127.31.74.212:34905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 03:04:54.308287 20217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:04:54.308616 20217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:04:54.308771 20217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:04:54.317126 20217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:04:54.317263 20217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:04:54.317422 20217 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.193
I20260806 03:04:54.328222 20217 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:34905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785985105255475-32043-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785985105255475-32043-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.74.193
--webserver_port=0
--tserver_master_addrs=127.31.74.254:36717
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.193
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785985105255475-32043-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:04:54.329800 20217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:04:54.332096 20217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:04:54.347735 20222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:04:54.347705 20225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:04:54.347829 20223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:04:54.349838 20217 server_base.cc:1034] running on GCE node
I20260806 03:04:54.350533 20217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:04:54.351673 20217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:04:54.352977 20217 hybrid_clock.cc:648] HybridClock initialized: now 1785985494352877 us; error 58 us; skew 500 ppm
I20260806 03:04:54.353449 20217 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 03:04:54.356743 20217 webserver.cc:460] Webserver started at http://127.31.74.193:37775/ using document root <none> and password file <none>
I20260806 03:04:54.357674 20217 fs_manager.cc:359] Metadata directory not provided
I20260806 03:04:54.357811 20217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:04:54.358191 20217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:04:54.361900 20217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785985105255475-32043-0/minicluster-data/ts-0/data/instance:
uuid: "7ac8fbda9348479e8861fd40904cd6fe"
format_stamp: "Formatted at 2026-08-06 03:04:54 on dist-test-slave-dhph"
server_key: "19e2e71b670302b3dabb753f767988dd"
server_key_iv: "84d4a077ca8c895a9bd4e6dea9006561"
server_key_version: "encryptionkey@0"
I20260806 03:04:54.363008 20217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785985105255475-32043-0/minicluster-data/ts-0/wal/instance:
uuid: "7ac8fbda9348479e8861fd40904cd6fe"
format_stamp: "Formatted at 2026-08-06 03:04:54 on dist-test-slave-dhph"
server_key: "19e2e71b670302b3dabb753f767988dd"
server_key_iv: "84d4a077ca8c895a9bd4e6dea9006561"
server_key_version: "encryptionkey@0"
I20260806 03:04:54.369383 20217 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260806 03:04:54.373528 20231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:04:54.375192 20217 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 03:04:54.375401 20217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785985105255475-32043-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785985105255475-32043-0/minicluster-data/ts-0/wal
uuid: "7ac8fbda9348479e8861fd40904cd6fe"
format_stamp: "Formatted at 2026-08-06 03:04:54 on dist-test-slave-dhph"
server_key: "19e2e71b670302b3dabb753f767988dd"
server_key_iv: "84d4a077ca8c895a9bd4e6dea9006561"
server_key_version: "encryptionkey@0"
I20260806 03:04:54.375705 20217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785985105255475-32043-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785985105255475-32043-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785985105255475-32043-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:04:54.395088 20217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:04:54.396483 20217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:04:54.396835 20217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:04:54.397921 20217 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:04:54.400316 20217 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:04:54.400470 20217 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:04:54.400601 20217 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:04:54.400689 20217 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:04:54.441092 20217 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.193:33205
I20260806 03:04:54.441322 20349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.193:33205 every 8 connection(s)
I20260806 03:04:54.442504 20217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785985105255475-32043-0/minicluster-data/ts-0/data/info.pb
I20260806 03:04:54.446645 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 20217
I20260806 03:04:54.446852 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785985105255475-32043-0/minicluster-data/ts-0/wal/instance
I20260806 03:04:54.447402 32043 external_mini_cluster.cc:1428] Setting key 33c8cd314d292899f0915f155c53a2f7
I20260806 03:04:54.450361 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785985105255475-32043-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785985105255475-32043-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785985105255475-32043-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.194:0
--local_ip_for_outbound_sockets=127.31.74.194
--webserver_interface=127.31.74.194
--webserver_port=0
--tserver_master_addrs=127.31.74.254:36717
--builtin_ntp_servers=127.31.74.212:34905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 03:04:54.456511 20352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:54.445576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.193:54935 (local address 127.31.74.254:36717)
0806 03:04:54.446088 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:54.446094 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:54.447350 (+ 1256us) server_negotiation.cc:407] Connection header received
0806 03:04:54.449477 (+ 2127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:54.449482 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:54.449569 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:54.449697 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:04:54.450945 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:54.451654 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:54.453200 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:54.453674 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:54.453858 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:54.455455 (+ 1597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:04:54.455461 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:04:54.455464 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:04:54.455486 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:04:54.455541 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:04:54.455555 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:04:54.455672 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:54.455979 (+ 307us) server_negotiation.cc:299] Negotiation successful
0806 03:04:54.456091 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":242,"threads_started":1}
I20260806 03:04:54.457598 20353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:54.445929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:36717 (local address 127.31.74.193:54935)
0806 03:04:54.447175 (+ 1246us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:54.447216 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 03:04:54.449173 (+ 1957us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:04:54.449954 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:04:54.449963 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:04:54.450054 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:04:54.450771 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:54.450791 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:54.451809 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:54.451818 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:54.453053 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:54.453065 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:54.453978 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:54.453988 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:54.454232 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:54.455062 (+ 830us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 03:04:54.455068 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:04:54.455129 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 03:04:54.455220 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:04:54.455672 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:04:54.455687 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:04:54.455711 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 03:04:54.455952 (+ 241us) client_negotiation.cc:240] Negotiation successful
0806 03:04:54.456402 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":381,"threads_started":1}
I20260806 03:04:54.459523 20350 heartbeater.cc:347] Connected to a master server at 127.31.74.254:36717
I20260806 03:04:54.460016 20350 heartbeater.cc:464] Registering TS with master...
I20260806 03:04:54.461210 20350 heartbeater.cc:511] Master 127.31.74.254:36717 requested a full tablet report, sending...
I20260806 03:04:54.463582 20156 ts_manager.cc:194] Registered new tserver with Master: 7ac8fbda9348479e8861fd40904cd6fe (127.31.74.193:33205)
I20260806 03:04:54.465418 20156 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.74.193:54935
W20260806 03:04:54.848203 20354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:04:54.848567 20354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:04:54.848717 20354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:04:54.856551 20354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:04:54.856712 20354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:04:54.856892 20354 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.194
I20260806 03:04:54.869480 20354 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:34905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785985105255475-32043-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785985105255475-32043-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.74.194
--webserver_port=0
--tserver_master_addrs=127.31.74.254:36717
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.194
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785985105255475-32043-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:04:54.870946 20354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:04:54.872838 20354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:04:54.889170 20359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:04:54.889321 20360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:04:54.889205 20362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:04:54.890988 20354 server_base.cc:1034] running on GCE node
I20260806 03:04:54.891755 20354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:04:54.892972 20354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:04:54.894263 20354 hybrid_clock.cc:648] HybridClock initialized: now 1785985494894166 us; error 56 us; skew 500 ppm
I20260806 03:04:54.894695 20354 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 03:04:54.897766 20354 webserver.cc:460] Webserver started at http://127.31.74.194:40999/ using document root <none> and password file <none>
I20260806 03:04:54.898679 20354 fs_manager.cc:359] Metadata directory not provided
I20260806 03:04:54.898819 20354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:04:54.899205 20354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:04:54.902683 20354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785985105255475-32043-0/minicluster-data/ts-1/data/instance:
uuid: "7527d8bc72a149268b77879aea7eab0c"
format_stamp: "Formatted at 2026-08-06 03:04:54 on dist-test-slave-dhph"
server_key: "a4e7123a93bad9560e40b4c3f6a90810"
server_key_iv: "e4512e54aedbc2375c092fa83df41c2a"
server_key_version: "encryptionkey@0"
I20260806 03:04:54.903800 20354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785985105255475-32043-0/minicluster-data/ts-1/wal/instance:
uuid: "7527d8bc72a149268b77879aea7eab0c"
format_stamp: "Formatted at 2026-08-06 03:04:54 on dist-test-slave-dhph"
server_key: "a4e7123a93bad9560e40b4c3f6a90810"
server_key_iv: "e4512e54aedbc2375c092fa83df41c2a"
server_key_version: "encryptionkey@0"
I20260806 03:04:54.910245 20354 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260806 03:04:54.915081 20368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:04:54.916692 20354 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 03:04:54.916940 20354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785985105255475-32043-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785985105255475-32043-0/minicluster-data/ts-1/wal
uuid: "7527d8bc72a149268b77879aea7eab0c"
format_stamp: "Formatted at 2026-08-06 03:04:54 on dist-test-slave-dhph"
server_key: "a4e7123a93bad9560e40b4c3f6a90810"
server_key_iv: "e4512e54aedbc2375c092fa83df41c2a"
server_key_version: "encryptionkey@0"
I20260806 03:04:54.917199 20354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785985105255475-32043-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785985105255475-32043-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785985105255475-32043-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:04:54.947325 20354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:04:54.948251 20354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:04:54.948599 20354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:04:54.949904 20354 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:04:54.951764 20354 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:04:54.951910 20354 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:04:54.952051 20354 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:04:54.952155 20354 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:04:55.008394 20487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.194:35395 every 8 connection(s)
I20260806 03:04:55.008394 20354 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.194:35395
I20260806 03:04:55.011092 20354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785985105255475-32043-0/minicluster-data/ts-1/data/info.pb
I20260806 03:04:55.020524 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 20354
I20260806 03:04:55.020752 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785985105255475-32043-0/minicluster-data/ts-1/wal/instance
I20260806 03:04:55.021482 32043 external_mini_cluster.cc:1428] Setting key 8ecd3810b990f37c246a9ee9dc83223a
I20260806 03:04:55.027015 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785985105255475-32043-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785985105255475-32043-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785985105255475-32043-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.74.195:0
--local_ip_for_outbound_sockets=127.31.74.195
--webserver_interface=127.31.74.195
--webserver_port=0
--tserver_master_addrs=127.31.74.254:36717
--builtin_ntp_servers=127.31.74.212:34905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 03:04:55.034806 20490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:55.014667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.194:54445 (local address 127.31.74.254:36717)
0806 03:04:55.015211 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:55.015217 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:55.016891 (+ 1674us) server_negotiation.cc:407] Connection header received
0806 03:04:55.019673 (+ 2782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:55.019680 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:55.019819 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:55.020012 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:04:55.022161 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:55.024058 (+ 1897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:55.026024 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:55.026688 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:55.026834 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:55.033804 (+ 6970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:04:55.033812 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:04:55.033816 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:04:55.033845 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:04:55.033894 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:04:55.033909 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:04:55.034012 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:55.034258 (+ 246us) server_negotiation.cc:299] Negotiation successful
0806 03:04:55.034400 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":302,"threads_started":1}
I20260806 03:04:55.036056 20491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:55.015510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:36717 (local address 127.31.74.194:54445)
0806 03:04:55.016693 (+ 1183us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:55.016740 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 03:04:55.019171 (+ 2431us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:04:55.020339 (+ 1168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:04:55.020351 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:04:55.020511 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:04:55.021504 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:55.021530 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:55.024282 (+ 2752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:55.024289 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:55.025856 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:55.025874 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:55.026937 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:55.026945 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:55.027262 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:55.033345 (+ 6083us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 03:04:55.033348 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:04:55.033404 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 03:04:55.033522 (+ 118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:04:55.034016 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:04:55.034031 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:04:55.034041 (+ 10us) client_negotiation.cc:769] Sending connection context
0806 03:04:55.034244 (+ 203us) client_negotiation.cc:240] Negotiation successful
0806 03:04:55.034770 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":675,"thread_start_us":403,"threads_started":1}
I20260806 03:04:55.037568 20488 heartbeater.cc:347] Connected to a master server at 127.31.74.254:36717
I20260806 03:04:55.037914 20488 heartbeater.cc:464] Registering TS with master...
I20260806 03:04:55.038864 20488 heartbeater.cc:511] Master 127.31.74.254:36717 requested a full tablet report, sending...
I20260806 03:04:55.040371 20156 ts_manager.cc:194] Registered new tserver with Master: 7527d8bc72a149268b77879aea7eab0c (127.31.74.194:35395)
I20260806 03:04:55.041225 20156 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.74.194:54445
W20260806 03:04:55.421507 20492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:04:55.421904 20492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:04:55.422039 20492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:04:55.430575 20492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:04:55.430755 20492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:04:55.430877 20492 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.74.195
I20260806 03:04:55.441169 20492 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:34905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785985105255475-32043-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785985105255475-32043-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.74.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.74.195
--webserver_port=0
--tserver_master_addrs=127.31.74.254:36717
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.74.195
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785985105255475-32043-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:04:55.442675 20492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:04:55.445000 20492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:04:55.461704 20498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:04:55.461668 20499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:04:55.461668 20501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:04:55.464754 20492 server_base.cc:1034] running on GCE node
I20260806 03:04:55.466269 20492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:04:55.467654 20492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:04:55.468935 20350 heartbeater.cc:503] Master 127.31.74.254:36717 was elected leader, sending a full tablet report...
I20260806 03:04:55.469013 20492 hybrid_clock.cc:648] HybridClock initialized: now 1785985495468947 us; error 59 us; skew 500 ppm
I20260806 03:04:55.469448 20492 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 03:04:55.472271 20492 webserver.cc:460] Webserver started at http://127.31.74.195:33533/ using document root <none> and password file <none>
I20260806 03:04:55.473317 20492 fs_manager.cc:359] Metadata directory not provided
I20260806 03:04:55.473429 20492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:04:55.473755 20492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:04:55.476944 20492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785985105255475-32043-0/minicluster-data/ts-2/data/instance:
uuid: "23a395b3ce0449eaa4bf45325313af4a"
format_stamp: "Formatted at 2026-08-06 03:04:55 on dist-test-slave-dhph"
server_key: "eac6e387a0ba63dd077de4a5a78bbba6"
server_key_iv: "562d3d14f245de3cd90729f5c0597ead"
server_key_version: "encryptionkey@0"
I20260806 03:04:55.477722 20492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785985105255475-32043-0/minicluster-data/ts-2/wal/instance:
uuid: "23a395b3ce0449eaa4bf45325313af4a"
format_stamp: "Formatted at 2026-08-06 03:04:55 on dist-test-slave-dhph"
server_key: "eac6e387a0ba63dd077de4a5a78bbba6"
server_key_iv: "562d3d14f245de3cd90729f5c0597ead"
server_key_version: "encryptionkey@0"
I20260806 03:04:55.486135 20492 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.010s sys 0.000s
I20260806 03:04:55.490547 20507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:04:55.492250 20492 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 03:04:55.492447 20492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785985105255475-32043-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785985105255475-32043-0/minicluster-data/ts-2/wal
uuid: "23a395b3ce0449eaa4bf45325313af4a"
format_stamp: "Formatted at 2026-08-06 03:04:55 on dist-test-slave-dhph"
server_key: "eac6e387a0ba63dd077de4a5a78bbba6"
server_key_iv: "562d3d14f245de3cd90729f5c0597ead"
server_key_version: "encryptionkey@0"
I20260806 03:04:55.492697 20492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785985105255475-32043-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785985105255475-32043-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785985105255475-32043-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:04:55.518265 20492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:04:55.519351 20492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:04:55.519732 20492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:04:55.521106 20492 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:04:55.523064 20492 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:04:55.523197 20492 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:04:55.523365 20492 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:04:55.523461 20492 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:04:55.569226 20492 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.195:35239
I20260806 03:04:55.569320 20624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.195:35239 every 8 connection(s)
I20260806 03:04:55.570864 20492 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785985105255475-32043-0/minicluster-data/ts-2/data/info.pb
I20260806 03:04:55.580583 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 20492
I20260806 03:04:55.580809 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785985105255475-32043-0/minicluster-data/ts-2/wal/instance
I20260806 03:04:55.581459 32043 external_mini_cluster.cc:1428] Setting key c0ecc9ad8a9049f72d57ce8f8da1918c
I20260806 03:04:55.588538 20627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:55.573737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.195:51131 (local address 127.31.74.254:36717)
0806 03:04:55.574222 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:55.574227 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:55.576604 (+ 2377us) server_negotiation.cc:407] Connection header received
0806 03:04:55.579336 (+ 2732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:55.579343 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:55.579464 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:55.579632 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:04:55.581315 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:55.582452 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:55.584911 (+ 2459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:55.585376 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:55.585522 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:55.587479 (+ 1957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:04:55.587485 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:04:55.587489 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:04:55.587511 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:04:55.587560 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:04:55.587584 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:04:55.587693 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:55.587964 (+ 271us) server_negotiation.cc:299] Negotiation successful
0806 03:04:55.588084 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":250,"threads_started":1}
I20260806 03:04:55.589684 20628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:55.574836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:36717 (local address 127.31.74.195:51131)
0806 03:04:55.576296 (+ 1460us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:55.576356 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0806 03:04:55.578879 (+ 2523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:04:55.579914 (+ 1035us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:04:55.579923 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:04:55.580036 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:04:55.581037 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:55.581069 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:55.582662 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:55.582668 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:55.584732 (+ 2064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:55.584747 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:55.585643 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:55.585652 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:55.585917 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:55.586945 (+ 1028us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 03:04:55.586948 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:04:55.586994 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 03:04:55.587195 (+ 201us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:04:55.587700 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:04:55.587716 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:04:55.587726 (+ 10us) client_negotiation.cc:769] Sending connection context
0806 03:04:55.587935 (+ 209us) client_negotiation.cc:240] Negotiation successful
0806 03:04:55.588424 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":760,"thread_start_us":431,"threads_started":1}
I20260806 03:04:55.591391 20625 heartbeater.cc:347] Connected to a master server at 127.31.74.254:36717
I20260806 03:04:55.591676 20625 heartbeater.cc:464] Registering TS with master...
I20260806 03:04:55.592449 20625 heartbeater.cc:511] Master 127.31.74.254:36717 requested a full tablet report, sending...
I20260806 03:04:55.593885 20156 ts_manager.cc:194] Registered new tserver with Master: 23a395b3ce0449eaa4bf45325313af4a (127.31.74.195:35239)
I20260806 03:04:55.594873 20156 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.74.195:51131
I20260806 03:04:55.598017 32043 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 03:04:55.612701 20627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:55.604899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41512 (local address 127.31.74.254:36717)
0806 03:04:55.605193 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:55.605204 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:55.605920 (+ 716us) server_negotiation.cc:407] Connection header received
0806 03:04:55.606352 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:55.606359 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:55.606461 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:55.606728 (+ 267us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:04:55.607483 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:55.608404 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:55.610025 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:55.610430 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:55.610536 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:55.611842 (+ 1306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:04:55.611847 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:04:55.611850 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:04:55.611872 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:04:55.611905 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:04:55.611919 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:04:55.611996 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:55.612266 (+ 270us) server_negotiation.cc:299] Negotiation successful
0806 03:04:55.612408 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260806 03:04:55.618768 20156 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41512:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 03:04:55.625905 20156 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 03:04:55.653223 20639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:55.640750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37676 (local address 127.31.74.194:35395)
0806 03:04:55.642034 (+ 1284us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:55.642043 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:55.642077 (+ 34us) server_negotiation.cc:407] Connection header received
0806 03:04:55.642302 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:55.642310 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:55.642519 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:55.642764 (+ 245us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:04:55.643963 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:55.645699 (+ 1736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:55.650300 (+ 4601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:55.651909 (+ 1609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:55.652101 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:55.652451 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:55.652632 (+ 181us) server_negotiation.cc:299] Negotiation successful
0806 03:04:55.652841 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1097,"thread_start_us":1040,"threads_started":1}
I20260806 03:04:55.654726 20638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:55.640502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.194:35395 (local address 127.0.0.1:37676)
0806 03:04:55.641820 (+ 1318us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:55.641871 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 03:04:55.642055 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:04:55.642787 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:04:55.642795 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:04:55.642828 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:04:55.643513 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:55.643545 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:55.645924 (+ 2379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:55.645930 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:55.650049 (+ 4119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:55.650066 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:55.652137 (+ 2071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:55.652144 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:55.652358 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:55.652411 (+ 53us) client_negotiation.cc:769] Sending connection context
0806 03:04:55.652569 (+ 158us) client_negotiation.cc:240] Negotiation successful
0806 03:04:55.654258 (+ 1689us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":983,"mutex_wait_us":1414,"thread_start_us":816,"threads_started":1}
I20260806 03:04:55.655563 20640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:55.642890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.195:35239 (local address 127.0.0.1:53992)
0806 03:04:55.644001 (+ 1111us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:55.644030 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0806 03:04:55.644165 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:04:55.647751 (+ 3586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:04:55.647758 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:04:55.647803 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:04:55.648611 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:55.648629 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:55.651051 (+ 2422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:55.651058 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:55.652703 (+ 1645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:55.652719 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:55.654737 (+ 2018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:55.654744 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:55.654954 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:55.654991 (+ 37us) client_negotiation.cc:769] Sending connection context
0806 03:04:55.655098 (+ 107us) client_negotiation.cc:240] Negotiation successful
0806 03:04:55.655242 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":962,"mutex_wait_us":2,"thread_start_us":630,"threads_started":1}
I20260806 03:04:55.655730 20641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:55.643499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53992 (local address 127.31.74.195:35239)
0806 03:04:55.645348 (+ 1849us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:55.645519 (+ 171us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:55.646129 (+ 610us) server_negotiation.cc:407] Connection header received
0806 03:04:55.646294 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:55.646307 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:55.647275 (+ 968us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:55.648894 (+ 1619us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:04:55.649247 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:55.650916 (+ 1669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:55.652958 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:55.654480 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:55.654661 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:55.654977 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:55.655172 (+ 195us) server_negotiation.cc:299] Negotiation successful
0806 03:04:55.655381 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1208,"thread_start_us":1795,"threads_started":1}
I20260806 03:04:55.659543 20642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:55.644430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43954 (local address 127.31.74.193:33205)
0806 03:04:55.646228 (+ 1798us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:55.646369 (+ 141us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:55.648701 (+ 2332us) server_negotiation.cc:407] Connection header received
0806 03:04:55.649056 (+ 355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:55.649065 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:55.649385 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:55.649573 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:04:55.650464 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:55.652036 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:55.656118 (+ 4082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:55.657769 (+ 1651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:55.658080 (+ 311us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:55.658430 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:55.658919 (+ 489us) server_negotiation.cc:299] Negotiation successful
0806 03:04:55.659152 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1593,"thread_start_us":2269,"threads_started":1}
I20260806 03:04:55.660153 20643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:55.646160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:33205 (local address 127.0.0.1:43954)
0806 03:04:55.648544 (+ 2384us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:55.648583 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 03:04:55.648835 (+ 252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:04:55.649706 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:04:55.649711 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:04:55.649744 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:04:55.650170 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:55.650182 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:55.652277 (+ 2095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:55.652282 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:55.655159 (+ 2877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:55.655173 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:55.658240 (+ 3067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:55.658254 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:55.658497 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:55.658535 (+ 38us) client_negotiation.cc:769] Sending connection context
0806 03:04:55.658672 (+ 137us) client_negotiation.cc:240] Negotiation successful
0806 03:04:55.659820 (+ 1148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2170,"mutex_wait_us":4,"thread_start_us":1363,"threads_started":1}
I20260806 03:04:55.659617 20557 tablet_service.cc:1467] Processing CreateTablet for tablet 62eb6a4dd40349348fcf6991816fe6c0 (DEFAULT_TABLE table=test-table [id=e503c8c41e414ef0b6cbc9a073ce0b54]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:04:55.659618 20421 tablet_service.cc:1467] Processing CreateTablet for tablet 62eb6a4dd40349348fcf6991816fe6c0 (DEFAULT_TABLE table=test-table [id=e503c8c41e414ef0b6cbc9a073ce0b54]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:04:55.661324 20421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62eb6a4dd40349348fcf6991816fe6c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:04:55.662941 20557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62eb6a4dd40349348fcf6991816fe6c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:04:55.664618 20282 tablet_service.cc:1467] Processing CreateTablet for tablet 62eb6a4dd40349348fcf6991816fe6c0 (DEFAULT_TABLE table=test-table [id=e503c8c41e414ef0b6cbc9a073ce0b54]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 03:04:55.668332 20282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62eb6a4dd40349348fcf6991816fe6c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:04:55.677168 20644 tablet_bootstrap.cc:492] T 62eb6a4dd40349348fcf6991816fe6c0 P 23a395b3ce0449eaa4bf45325313af4a: Bootstrap starting.
I20260806 03:04:55.682145 20645 tablet_bootstrap.cc:492] T 62eb6a4dd40349348fcf6991816fe6c0 P 7527d8bc72a149268b77879aea7eab0c: Bootstrap starting.
I20260806 03:04:55.684528 20644 tablet_bootstrap.cc:654] T 62eb6a4dd40349348fcf6991816fe6c0 P 23a395b3ce0449eaa4bf45325313af4a: Neither blocks nor log segments found. Creating new log.
I20260806 03:04:55.686306 20644 log.cc:824] T 62eb6a4dd40349348fcf6991816fe6c0 P 23a395b3ce0449eaa4bf45325313af4a: Log is configured to *not* fsync() on all Append() calls
I20260806 03:04:55.687959 20645 tablet_bootstrap.cc:654] T 62eb6a4dd40349348fcf6991816fe6c0 P 7527d8bc72a149268b77879aea7eab0c: Neither blocks nor log segments found. Creating new log.
I20260806 03:04:55.689060 20645 log.cc:824] T 62eb6a4dd40349348fcf6991816fe6c0 P 7527d8bc72a149268b77879aea7eab0c: Log is configured to *not* fsync() on all Append() calls
I20260806 03:04:55.690246 20644 tablet_bootstrap.cc:492] T 62eb6a4dd40349348fcf6991816fe6c0 P 23a395b3ce0449eaa4bf45325313af4a: No bootstrap required, opened a new log
I20260806 03:04:55.690681 20644 ts_tablet_manager.cc:1397] T 62eb6a4dd40349348fcf6991816fe6c0 P 23a395b3ce0449eaa4bf45325313af4a: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.006s
I20260806 03:04:55.691264 20649 tablet_bootstrap.cc:492] T 62eb6a4dd40349348fcf6991816fe6c0 P 7ac8fbda9348479e8861fd40904cd6fe: Bootstrap starting.
I20260806 03:04:55.692575 20645 tablet_bootstrap.cc:492] T 62eb6a4dd40349348fcf6991816fe6c0 P 7527d8bc72a149268b77879aea7eab0c: No bootstrap required, opened a new log
I20260806 03:04:55.692888 20645 ts_tablet_manager.cc:1397] T 62eb6a4dd40349348fcf6991816fe6c0 P 7527d8bc72a149268b77879aea7eab0c: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260806 03:04:55.695518 20649 tablet_bootstrap.cc:654] T 62eb6a4dd40349348fcf6991816fe6c0 P 7ac8fbda9348479e8861fd40904cd6fe: Neither blocks nor log segments found. Creating new log.
I20260806 03:04:55.697399 20649 log.cc:824] T 62eb6a4dd40349348fcf6991816fe6c0 P 7ac8fbda9348479e8861fd40904cd6fe: Log is configured to *not* fsync() on all Append() calls
I20260806 03:04:55.701192 20644 raft_consensus.cc:348] T 62eb6a4dd40349348fcf6991816fe6c0 P 23a395b3ce0449eaa4bf45325313af4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7527d8bc72a149268b77879aea7eab0c" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 35395 } } peers { permanent_uuid: "23a395b3ce0449eaa4bf45325313af4a" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 35239 } } peers { permanent_uuid: "7ac8fbda9348479e8861fd40904cd6fe" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 33205 } }
I20260806 03:04:55.701874 20649 tablet_bootstrap.cc:492] T 62eb6a4dd40349348fcf6991816fe6c0 P 7ac8fbda9348479e8861fd40904cd6fe: No bootstrap required, opened a new log
I20260806 03:04:55.702031 20644 raft_consensus.cc:374] T 62eb6a4dd40349348fcf6991816fe6c0 P 23a395b3ce0449eaa4bf45325313af4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:04:55.702184 20644 raft_consensus.cc:729] T 62eb6a4dd40349348fcf6991816fe6c0 P 23a395b3ce0449eaa4bf45325313af4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23a395b3ce0449eaa4bf45325313af4a, State: Initialized, Role: FOLLOWER
I20260806 03:04:55.702332 20649 ts_tablet_manager.cc:1397] T 62eb6a4dd40349348fcf6991816fe6c0 P 7ac8fbda9348479e8861fd40904cd6fe: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.005s
I20260806 03:04:55.703148 20644 consensus_queue.cc:260] T 62eb6a4dd40349348fcf6991816fe6c0 P 23a395b3ce0449eaa4bf45325313af4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7527d8bc72a149268b77879aea7eab0c" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 35395 } } peers { permanent_uuid: "23a395b3ce0449eaa4bf45325313af4a" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 35239 } } peers { permanent_uuid: "7ac8fbda9348479e8861fd40904cd6fe" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 33205 } }
I20260806 03:04:55.705132 20625 heartbeater.cc:503] Master 127.31.74.254:36717 was elected leader, sending a full tablet report...
I20260806 03:04:55.705937 20644 ts_tablet_manager.cc:1428] T 62eb6a4dd40349348fcf6991816fe6c0 P 23a395b3ce0449eaa4bf45325313af4a: Time spent starting tablet: real 0.015s user 0.012s sys 0.001s
I20260806 03:04:55.706272 20645 raft_consensus.cc:348] T 62eb6a4dd40349348fcf6991816fe6c0 P 7527d8bc72a149268b77879aea7eab0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7527d8bc72a149268b77879aea7eab0c" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 35395 } } peers { permanent_uuid: "23a395b3ce0449eaa4bf45325313af4a" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 35239 } } peers { permanent_uuid: "7ac8fbda9348479e8861fd40904cd6fe" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 33205 } }
I20260806 03:04:55.706820 20645 raft_consensus.cc:374] T 62eb6a4dd40349348fcf6991816fe6c0 P 7527d8bc72a149268b77879aea7eab0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:04:55.706962 20645 raft_consensus.cc:729] T 62eb6a4dd40349348fcf6991816fe6c0 P 7527d8bc72a149268b77879aea7eab0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7527d8bc72a149268b77879aea7eab0c, State: Initialized, Role: FOLLOWER
I20260806 03:04:55.707652 20645 consensus_queue.cc:260] T 62eb6a4dd40349348fcf6991816fe6c0 P 7527d8bc72a149268b77879aea7eab0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7527d8bc72a149268b77879aea7eab0c" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 35395 } } peers { permanent_uuid: "23a395b3ce0449eaa4bf45325313af4a" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 35239 } } peers { permanent_uuid: "7ac8fbda9348479e8861fd40904cd6fe" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 33205 } }
I20260806 03:04:55.709643 20645 ts_tablet_manager.cc:1428] T 62eb6a4dd40349348fcf6991816fe6c0 P 7527d8bc72a149268b77879aea7eab0c: Time spent starting tablet: real 0.017s user 0.017s sys 0.000s
I20260806 03:04:55.710819 20488 heartbeater.cc:503] Master 127.31.74.254:36717 was elected leader, sending a full tablet report...
I20260806 03:04:55.714211 20649 raft_consensus.cc:348] T 62eb6a4dd40349348fcf6991816fe6c0 P 7ac8fbda9348479e8861fd40904cd6fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7527d8bc72a149268b77879aea7eab0c" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 35395 } } peers { permanent_uuid: "23a395b3ce0449eaa4bf45325313af4a" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 35239 } } peers { permanent_uuid: "7ac8fbda9348479e8861fd40904cd6fe" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 33205 } }
I20260806 03:04:55.714653 20649 raft_consensus.cc:374] T 62eb6a4dd40349348fcf6991816fe6c0 P 7ac8fbda9348479e8861fd40904cd6fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:04:55.714778 20649 raft_consensus.cc:729] T 62eb6a4dd40349348fcf6991816fe6c0 P 7ac8fbda9348479e8861fd40904cd6fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ac8fbda9348479e8861fd40904cd6fe, State: Initialized, Role: FOLLOWER
I20260806 03:04:55.715413 20649 consensus_queue.cc:260] T 62eb6a4dd40349348fcf6991816fe6c0 P 7ac8fbda9348479e8861fd40904cd6fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7527d8bc72a149268b77879aea7eab0c" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 35395 } } peers { permanent_uuid: "23a395b3ce0449eaa4bf45325313af4a" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 35239 } } peers { permanent_uuid: "7ac8fbda9348479e8861fd40904cd6fe" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 33205 } }
I20260806 03:04:55.716997 20649 ts_tablet_manager.cc:1428] T 62eb6a4dd40349348fcf6991816fe6c0 P 7ac8fbda9348479e8861fd40904cd6fe: Time spent starting tablet: real 0.014s user 0.007s sys 0.007s
W20260806 03:04:55.764035 20489 tablet.cc:2369] T 62eb6a4dd40349348fcf6991816fe6c0 P 7527d8bc72a149268b77879aea7eab0c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 03:04:55.822932 20626 tablet.cc:2369] T 62eb6a4dd40349348fcf6991816fe6c0 P 23a395b3ce0449eaa4bf45325313af4a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 03:04:55.868014 20653 raft_consensus.cc:482] T 62eb6a4dd40349348fcf6991816fe6c0 P 7ac8fbda9348479e8861fd40904cd6fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:04:55.868404 20653 raft_consensus.cc:504] T 62eb6a4dd40349348fcf6991816fe6c0 P 7ac8fbda9348479e8861fd40904cd6fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7527d8bc72a149268b77879aea7eab0c" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 35395 } } peers { permanent_uuid: "23a395b3ce0449eaa4bf45325313af4a" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 35239 } } peers { permanent_uuid: "7ac8fbda9348479e8861fd40904cd6fe" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 33205 } }
I20260806 03:04:55.870738 20653 leader_election.cc:288] T 62eb6a4dd40349348fcf6991816fe6c0 P 7ac8fbda9348479e8861fd40904cd6fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7527d8bc72a149268b77879aea7eab0c (127.31.74.194:35395), 23a395b3ce0449eaa4bf45325313af4a (127.31.74.195:35239)
I20260806 03:04:55.879464 20639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:55.871586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.193:54777 (local address 127.31.74.194:35395)
0806 03:04:55.871867 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:55.871873 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:55.872761 (+ 888us) server_negotiation.cc:407] Connection header received
0806 03:04:55.873108 (+ 347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:55.873120 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:55.873249 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:55.873319 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:04:55.874086 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:55.875194 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:55.877661 (+ 2467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:55.878458 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:55.878665 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:55.878743 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:55.879016 (+ 273us) server_negotiation.cc:299] Negotiation successful
0806 03:04:55.879149 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260806 03:04:55.879550 20654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:55.871500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.194:35395 (local address 127.31.74.193:54777)
0806 03:04:55.872540 (+ 1040us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:55.872562 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 03:04:55.872786 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:04:55.873543 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:04:55.873546 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:04:55.873569 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:04:55.873927 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:55.873935 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:55.875367 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:55.875375 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:55.877403 (+ 2028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:55.877432 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:55.878685 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:55.878691 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:55.878850 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:55.878875 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 03:04:55.878977 (+ 102us) client_negotiation.cc:240] Negotiation successful
0806 03:04:55.879120 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":889,"mutex_wait_us":29,"spinlock_wait_cycles":48384,"thread_start_us":699,"threads_started":1}
I20260806 03:04:55.879882 20641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:55.873155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.74.193:42819 (local address 127.31.74.195:35239)
0806 03:04:55.873375 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:55.873383 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:55.873661 (+ 278us) server_negotiation.cc:407] Connection header received
0806 03:04:55.873938 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:55.873942 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:55.874002 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:55.874101 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 03:04:55.874695 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:55.875394 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:55.877665 (+ 2271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:55.878810 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:55.879034 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:55.879118 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:55.879474 (+ 356us) server_negotiation.cc:299] Negotiation successful
0806 03:04:55.879616 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260806 03:04:55.880033 20656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:55.872795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.195:35239 (local address 127.31.74.193:42819)
0806 03:04:55.873541 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:55.873557 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 03:04:55.873753 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:04:55.874116 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:04:55.874121 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:04:55.874144 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 03:04:55.874490 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:55.874499 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:55.875532 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:55.875539 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:55.877400 (+ 1861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:04:55.877416 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:04:55.879034 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:04:55.879042 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:04:55.879264 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:55.879322 (+ 58us) client_negotiation.cc:769] Sending connection context
0806 03:04:55.879445 (+ 123us) client_negotiation.cc:240] Negotiation successful
0806 03:04:55.879608 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"mutex_wait_us":3,"thread_start_us":374,"threads_started":1}
I20260806 03:04:55.881096 20579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62eb6a4dd40349348fcf6991816fe6c0" candidate_uuid: "7ac8fbda9348479e8861fd40904cd6fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23a395b3ce0449eaa4bf45325313af4a" is_pre_election: true
I20260806 03:04:55.881098 20442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62eb6a4dd40349348fcf6991816fe6c0" candidate_uuid: "7ac8fbda9348479e8861fd40904cd6fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7527d8bc72a149268b77879aea7eab0c" is_pre_election: true
I20260806 03:04:55.881760 20442 raft_consensus.cc:2454] T 62eb6a4dd40349348fcf6991816fe6c0 P 7527d8bc72a149268b77879aea7eab0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ac8fbda9348479e8861fd40904cd6fe in term 0.
I20260806 03:04:55.881760 20579 raft_consensus.cc:2454] T 62eb6a4dd40349348fcf6991816fe6c0 P 23a395b3ce0449eaa4bf45325313af4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ac8fbda9348479e8861fd40904cd6fe in term 0.
I20260806 03:04:55.882484 20232 leader_election.cc:302] T 62eb6a4dd40349348fcf6991816fe6c0 P 7ac8fbda9348479e8861fd40904cd6fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23a395b3ce0449eaa4bf45325313af4a, 7ac8fbda9348479e8861fd40904cd6fe; no voters:
I20260806 03:04:55.882929 20653 raft_consensus.cc:2781] T 62eb6a4dd40349348fcf6991816fe6c0 P 7ac8fbda9348479e8861fd40904cd6fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:04:55.883111 20653 raft_consensus.cc:482] T 62eb6a4dd40349348fcf6991816fe6c0 P 7ac8fbda9348479e8861fd40904cd6fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:04:55.883237 20653 raft_consensus.cc:3037] T 62eb6a4dd40349348fcf6991816fe6c0 P 7ac8fbda9348479e8861fd40904cd6fe [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:04:55.885715 20653 raft_consensus.cc:504] T 62eb6a4dd40349348fcf6991816fe6c0 P 7ac8fbda9348479e8861fd40904cd6fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7527d8bc72a149268b77879aea7eab0c" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 35395 } } peers { permanent_uuid: "23a395b3ce0449eaa4bf45325313af4a" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 35239 } } peers { permanent_uuid: "7ac8fbda9348479e8861fd40904cd6fe" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 33205 } }
I20260806 03:04:55.886425 20653 leader_election.cc:288] T 62eb6a4dd40349348fcf6991816fe6c0 P 7ac8fbda9348479e8861fd40904cd6fe [CANDIDATE]: Term 1 election: Requested vote from peers 7527d8bc72a149268b77879aea7eab0c (127.31.74.194:35395), 23a395b3ce0449eaa4bf45325313af4a (127.31.74.195:35239)
I20260806 03:04:55.887219 20442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62eb6a4dd40349348fcf6991816fe6c0" candidate_uuid: "7ac8fbda9348479e8861fd40904cd6fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7527d8bc72a149268b77879aea7eab0c"
I20260806 03:04:55.887280 20579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62eb6a4dd40349348fcf6991816fe6c0" candidate_uuid: "7ac8fbda9348479e8861fd40904cd6fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23a395b3ce0449eaa4bf45325313af4a"
I20260806 03:04:55.887465 20442 raft_consensus.cc:3037] T 62eb6a4dd40349348fcf6991816fe6c0 P 7527d8bc72a149268b77879aea7eab0c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:04:55.887566 20579 raft_consensus.cc:3037] T 62eb6a4dd40349348fcf6991816fe6c0 P 23a395b3ce0449eaa4bf45325313af4a [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:04:55.890566 20579 raft_consensus.cc:2454] T 62eb6a4dd40349348fcf6991816fe6c0 P 23a395b3ce0449eaa4bf45325313af4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ac8fbda9348479e8861fd40904cd6fe in term 1.
I20260806 03:04:55.891464 20232 leader_election.cc:302] T 62eb6a4dd40349348fcf6991816fe6c0 P 7ac8fbda9348479e8861fd40904cd6fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23a395b3ce0449eaa4bf45325313af4a, 7ac8fbda9348479e8861fd40904cd6fe; no voters:
I20260806 03:04:55.892175 20653 raft_consensus.cc:2781] T 62eb6a4dd40349348fcf6991816fe6c0 P 7ac8fbda9348479e8861fd40904cd6fe [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:04:55.892119 20442 raft_consensus.cc:2454] T 62eb6a4dd40349348fcf6991816fe6c0 P 7527d8bc72a149268b77879aea7eab0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ac8fbda9348479e8861fd40904cd6fe in term 1.
I20260806 03:04:55.893101 20653 raft_consensus.cc:686] T 62eb6a4dd40349348fcf6991816fe6c0 P 7ac8fbda9348479e8861fd40904cd6fe [term 1 LEADER]: Becoming Leader. State: Replica: 7ac8fbda9348479e8861fd40904cd6fe, State: Running, Role: LEADER
I20260806 03:04:55.893682 20653 consensus_queue.cc:237] T 62eb6a4dd40349348fcf6991816fe6c0 P 7ac8fbda9348479e8861fd40904cd6fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7527d8bc72a149268b77879aea7eab0c" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 35395 } } peers { permanent_uuid: "23a395b3ce0449eaa4bf45325313af4a" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 35239 } } peers { permanent_uuid: "7ac8fbda9348479e8861fd40904cd6fe" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 33205 } }
I20260806 03:04:55.900146 20154 catalog_manager.cc:5495] T 62eb6a4dd40349348fcf6991816fe6c0 P 7ac8fbda9348479e8861fd40904cd6fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ac8fbda9348479e8861fd40904cd6fe (127.31.74.193). New cstate: current_term: 1 leader_uuid: "7ac8fbda9348479e8861fd40904cd6fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7527d8bc72a149268b77879aea7eab0c" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 35395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23a395b3ce0449eaa4bf45325313af4a" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 35239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ac8fbda9348479e8861fd40904cd6fe" member_type: VOTER last_known_addr { host: "127.31.74.193" port: 33205 } health_report { overall_health: HEALTHY } } }
W20260806 03:04:55.945684 20351 tablet.cc:2369] T 62eb6a4dd40349348fcf6991816fe6c0 P 7ac8fbda9348479e8861fd40904cd6fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 03:04:55.965512 20642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:55.958682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43968 (local address 127.31.74.193:33205)
0806 03:04:55.958947 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:55.958955 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:55.959022 (+ 67us) server_negotiation.cc:407] Connection header received
0806 03:04:55.959327 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:55.959332 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:55.959454 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:55.959614 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 03:04:55.960105 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:55.960902 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:55.962785 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:55.963304 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:55.963508 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:55.964078 (+ 570us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 03:04:55.964663 (+ 585us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 03:04:55.964861 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:55.965036 (+ 175us) server_negotiation.cc:299] Negotiation successful
0806 03:04:55.965168 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260806 03:04:55.976707 20442 raft_consensus.cc:1261] T 62eb6a4dd40349348fcf6991816fe6c0 P 7527d8bc72a149268b77879aea7eab0c [term 1 FOLLOWER]: Refusing update from remote peer 7ac8fbda9348479e8861fd40904cd6fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:04:55.976755 20579 raft_consensus.cc:1261] T 62eb6a4dd40349348fcf6991816fe6c0 P 23a395b3ce0449eaa4bf45325313af4a [term 1 FOLLOWER]: Refusing update from remote peer 7ac8fbda9348479e8861fd40904cd6fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:04:55.978785 20658 consensus_queue.cc:1035] T 62eb6a4dd40349348fcf6991816fe6c0 P 7ac8fbda9348479e8861fd40904cd6fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "7527d8bc72a149268b77879aea7eab0c" member_type: VOTER last_known_addr { host: "127.31.74.194" port: 35395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:04:55.979252 20653 consensus_queue.cc:1035] T 62eb6a4dd40349348fcf6991816fe6c0 P 7ac8fbda9348479e8861fd40904cd6fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "23a395b3ce0449eaa4bf45325313af4a" member_type: VOTER last_known_addr { host: "127.31.74.195" port: 35239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:04:56.025451 20154 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41512:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 03:04:56.025765 20154 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41512:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 03:04:56.030548 20154 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 52cc38329aeb4a229d0672ede00c2da9: Sending DeleteTablet for 3 replicas of tablet 62eb6a4dd40349348fcf6991816fe6c0
I20260806 03:04:56.032189 20421 tablet_service.cc:1514] Processing DeleteTablet for tablet 62eb6a4dd40349348fcf6991816fe6c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:04:56 UTC) from {username='slave'} at 127.0.0.1:37676
I20260806 03:04:56.032873 20557 tablet_service.cc:1514] Processing DeleteTablet for tablet 62eb6a4dd40349348fcf6991816fe6c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:04:56 UTC) from {username='slave'} at 127.0.0.1:53992
I20260806 03:04:56.032963 20282 tablet_service.cc:1514] Processing DeleteTablet for tablet 62eb6a4dd40349348fcf6991816fe6c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 03:04:56 UTC) from {username='slave'} at 127.0.0.1:43954
I20260806 03:04:56.034966 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 20217
I20260806 03:04:56.052663 20675 tablet_replica.cc:330] T 62eb6a4dd40349348fcf6991816fe6c0 P 7527d8bc72a149268b77879aea7eab0c: stopping tablet replica
I20260806 03:04:56.053510 20675 raft_consensus.cc:2229] T 62eb6a4dd40349348fcf6991816fe6c0 P 7527d8bc72a149268b77879aea7eab0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:04:56.054908 20677 tablet_replica.cc:330] T 62eb6a4dd40349348fcf6991816fe6c0 P 7ac8fbda9348479e8861fd40904cd6fe: stopping tablet replica
I20260806 03:04:56.055019 20675 raft_consensus.cc:2258] T 62eb6a4dd40349348fcf6991816fe6c0 P 7527d8bc72a149268b77879aea7eab0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:04:56.055614 20677 raft_consensus.cc:2229] T 62eb6a4dd40349348fcf6991816fe6c0 P 7ac8fbda9348479e8861fd40904cd6fe [term 1 LEADER]: Raft consensus shutting down.
I20260806 03:04:56.056550 20677 raft_consensus.cc:2258] T 62eb6a4dd40349348fcf6991816fe6c0 P 7ac8fbda9348479e8861fd40904cd6fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:04:56.058844 20675 ts_tablet_manager.cc:1905] T 62eb6a4dd40349348fcf6991816fe6c0 P 7527d8bc72a149268b77879aea7eab0c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:04:56.059027 20676 tablet_replica.cc:330] T 62eb6a4dd40349348fcf6991816fe6c0 P 23a395b3ce0449eaa4bf45325313af4a: stopping tablet replica
I20260806 03:04:56.059551 20676 raft_consensus.cc:2229] T 62eb6a4dd40349348fcf6991816fe6c0 P 23a395b3ce0449eaa4bf45325313af4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:04:56.060086 20676 raft_consensus.cc:2258] T 62eb6a4dd40349348fcf6991816fe6c0 P 23a395b3ce0449eaa4bf45325313af4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:04:56.060763 20677 ts_tablet_manager.cc:1905] T 62eb6a4dd40349348fcf6991816fe6c0 P 7ac8fbda9348479e8861fd40904cd6fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:04:56.062342 20676 ts_tablet_manager.cc:1905] T 62eb6a4dd40349348fcf6991816fe6c0 P 23a395b3ce0449eaa4bf45325313af4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 03:04:56.067431 20642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:56.036559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43982 (local address 127.31.74.193:33205)
0806 03:04:56.037114 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:56.037123 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:56.058180 (+ 21057us) server_negotiation.cc:407] Connection header received
0806 03:04:56.058417 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:56.058425 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:56.058562 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:56.058705 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:04:56.059507 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:56.060640 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:56.062708 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:56.063416 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:56.063544 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:56.065287 (+ 1743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:04:56.065299 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:04:56.065318 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:04:56.066215 (+ 897us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:04:56.066302 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:04:56.066356 (+ 54us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:04:56.066467 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:56.066809 (+ 342us) server_negotiation.cc:299] Negotiation successful
0806 03:04:56.066938 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 03:04:56.068712 20675 ts_tablet_manager.cc:1918] T 62eb6a4dd40349348fcf6991816fe6c0 P 7527d8bc72a149268b77879aea7eab0c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 03:04:56.069128 20675 log.cc:1196] T 62eb6a4dd40349348fcf6991816fe6c0 P 7527d8bc72a149268b77879aea7eab0c: Deleting WAL directory at /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785985105255475-32043-0/minicluster-data/ts-1/wal/wals/62eb6a4dd40349348fcf6991816fe6c0
I20260806 03:04:56.069213 20345 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 03:04:56.070771 20675 ts_tablet_manager.cc:1939] T 62eb6a4dd40349348fcf6991816fe6c0 P 7527d8bc72a149268b77879aea7eab0c: Deleting consensus metadata
I20260806 03:04:56.071528 20676 ts_tablet_manager.cc:1918] T 62eb6a4dd40349348fcf6991816fe6c0 P 23a395b3ce0449eaa4bf45325313af4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 03:04:56.071872 20676 log.cc:1196] T 62eb6a4dd40349348fcf6991816fe6c0 P 23a395b3ce0449eaa4bf45325313af4a: Deleting WAL directory at /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785985105255475-32043-0/minicluster-data/ts-2/wal/wals/62eb6a4dd40349348fcf6991816fe6c0
I20260806 03:04:56.072865 20676 ts_tablet_manager.cc:1939] T 62eb6a4dd40349348fcf6991816fe6c0 P 23a395b3ce0449eaa4bf45325313af4a: Deleting consensus metadata
I20260806 03:04:56.075138 20140 catalog_manager.cc:4841] TS 7527d8bc72a149268b77879aea7eab0c (127.31.74.194:35395): tablet 62eb6a4dd40349348fcf6991816fe6c0 (table test-table [id=e503c8c41e414ef0b6cbc9a073ce0b54]) successfully deleted
I20260806 03:04:56.075773 20140 catalog_manager.cc:4841] TS 23a395b3ce0449eaa4bf45325313af4a (127.31.74.195:35239): tablet 62eb6a4dd40349348fcf6991816fe6c0 (table test-table [id=e503c8c41e414ef0b6cbc9a073ce0b54]) successfully deleted
I20260806 03:04:56.187896 20677 ts_tablet_manager.cc:1918] T 62eb6a4dd40349348fcf6991816fe6c0 P 7ac8fbda9348479e8861fd40904cd6fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 03:04:56.188220 20677 log.cc:1196] T 62eb6a4dd40349348fcf6991816fe6c0 P 7ac8fbda9348479e8861fd40904cd6fe: Deleting WAL directory at /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785985105255475-32043-0/minicluster-data/ts-0/wal/wals/62eb6a4dd40349348fcf6991816fe6c0
I20260806 03:04:56.188670 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 20217
W20260806 03:04:56.218161 20140 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.74.193:33205 (error 108)
I20260806 03:04:56.218286 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 20354
I20260806 03:04:56.221091 20681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:56.219571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:33205 (local address 127.0.0.1:43986)
0806 03:04:56.220635 (+ 1064us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:56.220755 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.193:33205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":951,"thread_start_us":873,"threads_started":1}
W20260806 03:04:56.221403 20140 catalog_manager.cc:4584] TS 7ac8fbda9348479e8861fd40904cd6fe (127.31.74.193:33205): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 62eb6a4dd40349348fcf6991816fe6c0: Network error: Client connection negotiation failed: client connection to 127.31.74.193:33205: connect: Connection refused (error 111)
I20260806 03:04:56.227294 20639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:56.219696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37692 (local address 127.31.74.194:35395)
0806 03:04:56.219943 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:56.219948 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:56.219969 (+ 21us) server_negotiation.cc:407] Connection header received
0806 03:04:56.220033 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:56.220039 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:56.220135 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:56.220242 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:04:56.220892 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:56.222003 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:56.223713 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:56.224185 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:56.224349 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:56.225571 (+ 1222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:04:56.225582 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:04:56.225597 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:04:56.226325 (+ 728us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:04:56.226397 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:04:56.226421 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:04:56.226579 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:56.226756 (+ 177us) server_negotiation.cc:299] Negotiation successful
0806 03:04:56.226897 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260806 03:04:56.228132 20483 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 03:04:56.275667 20681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:56.275040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:33205 (local address 127.0.0.1:43998)
0806 03:04:56.275348 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:56.275472 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.193:33205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260806 03:04:56.276942 20681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:56.276478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:33205 (local address 127.0.0.1:44004)
0806 03:04:56.276671 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:56.276766 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.193:33205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260806 03:04:56.338521 20681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:56.337502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:33205 (local address 127.0.0.1:44006)
0806 03:04:56.338091 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:56.338288 (+ 197us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.193:33205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192}
I20260806 03:04:56.340674 20681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:56.339806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:33205 (local address 127.0.0.1:44016)
0806 03:04:56.340185 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:56.340362 (+ 177us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.193:33205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
I20260806 03:04:56.341919 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 20354
I20260806 03:04:56.367345 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 20492
I20260806 03:04:56.376927 20641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:56.368239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54006 (local address 127.31.74.195:35239)
0806 03:04:56.368515 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:56.368521 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:56.368768 (+ 247us) server_negotiation.cc:407] Connection header received
0806 03:04:56.369073 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:56.369079 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:56.369234 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:56.369388 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:04:56.370072 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:56.371240 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:56.373026 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:56.373503 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:56.373633 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:56.375163 (+ 1530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:04:56.375172 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:04:56.375188 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:04:56.375809 (+ 621us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:04:56.375862 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:04:56.375885 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:04:56.375994 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:56.376416 (+ 422us) server_negotiation.cc:299] Negotiation successful
0806 03:04:56.376532 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260806 03:04:56.377611 20620 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 03:04:56.422230 20681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:56.421593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:33205 (local address 127.0.0.1:44028)
0806 03:04:56.421865 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:56.421971 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.193:33205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260806 03:04:56.423720 20681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:56.423196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:33205 (local address 127.0.0.1:44030)
0806 03:04:56.423475 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:56.423557 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.193:33205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165}
I20260806 03:04:56.487121 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 20492
I20260806 03:04:56.507092 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 20124
I20260806 03:04:56.508528 20187 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 03:04:56.653695 20681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:56.651776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.193:33205 (local address 127.0.0.1:44034)
0806 03:04:56.652665 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0806 03:04:56.653414 (+ 749us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.193:33205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
I20260806 03:04:56.653834 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 20124
2026-08-06T03:04:56Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (3482 ms)
[----------] 6 tests from EncryptionPolicyTest (26909 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785985105255475-32043-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 03:04:56 dist-test-slave-dhph krb5kdc[20687](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 03:04:56 dist-test-slave-dhph krb5kdc[20687](info): set up 2 sockets
Aug 06 03:04:56 dist-test-slave-dhph krb5kdc[20687](info): commencing operation
krb5kdc: starting...
W20260806 03:04:57.939869 32043 mini_kdc.cc:121] Time spent starting KDC: real 1.231s user 0.000s sys 0.063s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785985105255475-32043-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785985105255475-32043-0/krb5kdc/test-user.keytab.
Aug 06 03:04:58 dist-test-slave-dhph krb5kdc[20687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T03:04:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:04:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785985105255475-32043-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785985105255475-32043-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 03:04:58.324141 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785985105255475-32043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785985105255475-32043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785985105255475-32043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38839
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785985105255475-32043-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38839
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:04:58.729552 20701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:04:58.729930 20701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:04:58.730091 20701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:04:58.738448 20701 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:04:58.738698 20701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:04:58.738821 20701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:04:58.738937 20701 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:04:58.739017 20701 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:04:58.750136 20701 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785985105255475-32043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785985105255475-32043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38839
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785985105255475-32043-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785985105255475-32043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:04:58.751837 20701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:04:58.753403 20701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:04:58.770020 20709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:04:58.769971 20707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:04:58.770071 20706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:04:58.776367 20701 server_base.cc:1034] running on GCE node
I20260806 03:04:58.777704 20701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:04:58.779621 20701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:04:58.780952 20701 hybrid_clock.cc:648] HybridClock initialized: now 1785985498780877 us; error 43 us; skew 500 ppm
I20260806 03:04:58.781646 20701 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 03:04:58 dist-test-slave-dhph krb5kdc[20687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985498, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:04:58.786052 20701 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 03:04:58.788115 20701 webserver.cc:460] Webserver started at http://127.0.0.1:45451/ using document root <none> and password file <none>
I20260806 03:04:58.789007 20701 fs_manager.cc:359] Metadata directory not provided
I20260806 03:04:58.789207 20701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:04:58.789606 20701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:04:58.793615 20701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785985105255475-32043-0/minicluster-data/master-0/data/instance:
uuid: "8ef165ef0d224817bb651983ada31be0"
format_stamp: "Formatted at 2026-08-06 03:04:58 on dist-test-slave-dhph"
server_key: "11ecd4aae59ae99f0c9421da53d10997"
server_key_iv: "b02720ed1866de7e959c19fc3efda588"
server_key_version: "encryptionkey@0"
I20260806 03:04:58.794677 20701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785985105255475-32043-0/minicluster-data/master-0/wal/instance:
uuid: "8ef165ef0d224817bb651983ada31be0"
format_stamp: "Formatted at 2026-08-06 03:04:58 on dist-test-slave-dhph"
server_key: "11ecd4aae59ae99f0c9421da53d10997"
server_key_iv: "b02720ed1866de7e959c19fc3efda588"
server_key_version: "encryptionkey@0"
I20260806 03:04:58.801107 20701 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260806 03:04:58.806073 20716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:04:58.808319 20701 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260806 03:04:58.808595 20701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785985105255475-32043-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785985105255475-32043-0/minicluster-data/master-0/wal
uuid: "8ef165ef0d224817bb651983ada31be0"
format_stamp: "Formatted at 2026-08-06 03:04:58 on dist-test-slave-dhph"
server_key: "11ecd4aae59ae99f0c9421da53d10997"
server_key_iv: "b02720ed1866de7e959c19fc3efda588"
server_key_version: "encryptionkey@0"
I20260806 03:04:58.808809 20701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785985105255475-32043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785985105255475-32043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785985105255475-32043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:04:58.829422 20701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:04:58.834445 20701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:04:58.834823 20701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:04:58.860531 20701 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38839
I20260806 03:04:58.860577 20771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38839 every 8 connection(s)
I20260806 03:04:58.862159 20701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
I20260806 03:04:58.865341 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 20701
I20260806 03:04:58.865600 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785985105255475-32043-0/minicluster-data/master-0/wal/instance
I20260806 03:04:58.866602 32043 external_mini_cluster.cc:1428] Setting key 3bc6fe80cfb0c3b526be0bf079fb23bd
I20260806 03:04:58.867437 20772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 06 03:04:58 dist-test-slave-dhph krb5kdc[20687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 03:04:58.885557 20772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ef165ef0d224817bb651983ada31be0: Bootstrap starting.
I20260806 03:04:58.891373 20772 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ef165ef0d224817bb651983ada31be0: Neither blocks nor log segments found. Creating new log.
I20260806 03:04:58.892979 20772 log.cc:824] T 00000000000000000000000000000000 P 8ef165ef0d224817bb651983ada31be0: Log is configured to *not* fsync() on all Append() calls
I20260806 03:04:58.896907 20772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ef165ef0d224817bb651983ada31be0: No bootstrap required, opened a new log
I20260806 03:04:58.897557 20775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:58.869250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32867 (local address 127.0.0.1:38839)
0806 03:04:58.870218 (+ 968us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:58.870231 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:58.870276 (+ 45us) server_negotiation.cc:407] Connection header received
0806 03:04:58.872172 (+ 1896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:58.872197 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:58.872676 (+ 479us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:58.873081 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:04:58.875382 (+ 2301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:58.876880 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:58.878997 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:58.879791 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:58.879950 (+ 159us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:58.885456 (+ 5506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:04:58.885469 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:04:58.885493 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:04:58.885560 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:04:58.891155 (+ 5595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:04:58.892743 (+ 1588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:04:58.892754 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:04:58.892766 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:04:58.893080 (+ 314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:04:58.893947 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:04:58.893951 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:04:58.893953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:04:58.894908 (+ 955us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:04:58.895257 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:58.895791 (+ 534us) server_negotiation.cc:299] Negotiation successful
0806 03:04:58.896160 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":737,"thread_start_us":478,"threads_started":1}
I20260806 03:04:58.903434 32043 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 03:04:58.907199 20772 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ef165ef0d224817bb651983ada31be0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ef165ef0d224817bb651983ada31be0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38839 } }
I20260806 03:04:58.907774 20772 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ef165ef0d224817bb651983ada31be0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:04:58.907917 20772 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ef165ef0d224817bb651983ada31be0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ef165ef0d224817bb651983ada31be0, State: Initialized, Role: FOLLOWER
I20260806 03:04:58.908569 20772 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ef165ef0d224817bb651983ada31be0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ef165ef0d224817bb651983ada31be0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38839 } }
I20260806 03:04:58.908871 20772 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ef165ef0d224817bb651983ada31be0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:04:58.909471 20772 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ef165ef0d224817bb651983ada31be0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:04:58.909780 20772 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ef165ef0d224817bb651983ada31be0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:04:58.914436 20772 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ef165ef0d224817bb651983ada31be0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ef165ef0d224817bb651983ada31be0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38839 } }
I20260806 03:04:58.915289 20772 leader_election.cc:302] T 00000000000000000000000000000000 P 8ef165ef0d224817bb651983ada31be0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ef165ef0d224817bb651983ada31be0; no voters:
I20260806 03:04:58.916086 20772 leader_election.cc:288] T 00000000000000000000000000000000 P 8ef165ef0d224817bb651983ada31be0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:04:58.916225 20782 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ef165ef0d224817bb651983ada31be0 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:04:58.917161 20782 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ef165ef0d224817bb651983ada31be0 [term 1 LEADER]: Becoming Leader. State: Replica: 8ef165ef0d224817bb651983ada31be0, State: Running, Role: LEADER
I20260806 03:04:58.918087 20782 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ef165ef0d224817bb651983ada31be0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ef165ef0d224817bb651983ada31be0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38839 } }
I20260806 03:04:58.921254 20782 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ef165ef0d224817bb651983ada31be0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ef165ef0d224817bb651983ada31be0. Latest consensus state: current_term: 1 leader_uuid: "8ef165ef0d224817bb651983ada31be0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ef165ef0d224817bb651983ada31be0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38839 } } }
I20260806 03:04:58.921720 20782 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ef165ef0d224817bb651983ada31be0 [sys.catalog]: This master's current role is: LEADER
I20260806 03:04:58.921880 20784 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ef165ef0d224817bb651983ada31be0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ef165ef0d224817bb651983ada31be0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ef165ef0d224817bb651983ada31be0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38839 } } }
I20260806 03:04:58.925810 20784 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ef165ef0d224817bb651983ada31be0 [sys.catalog]: This master's current role is: LEADER
I20260806 03:04:58.927044 20772 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ef165ef0d224817bb651983ada31be0 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:04:58.927430 20786 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:04:58.935994 20775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:04:58.909315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53283 (local address 127.0.0.1:38839)
0806 03:04:58.909591 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0806 03:04:58.909602 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 03:04:58.910071 (+ 469us) server_negotiation.cc:407] Connection header received
0806 03:04:58.910819 (+ 748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:04:58.910825 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:04:58.910905 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:04:58.911010 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:04:58.913371 (+ 2361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:58.914402 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:58.916446 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:04:58.917470 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:04:58.918384 (+ 914us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:04:58.921597 (+ 3213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:04:58.921604 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:04:58.921610 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:04:58.921662 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:04:58.930368 (+ 8706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:04:58.931786 (+ 1418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:04:58.931795 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:04:58.931802 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:04:58.932177 (+ 375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:04:58.933004 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:04:58.933011 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:04:58.933017 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:04:58.933484 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:04:58.933850 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0806 03:04:58.935300 (+ 1450us) server_negotiation.cc:299] Negotiation successful
0806 03:04:58.935540 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260806 03:04:58.939138 20786 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:04:58.951907 20786 catalog_manager.cc:1347] Generated new cluster ID: 781d945e6b854982885d855b5eaa711b
I20260806 03:04:58.952047 20786 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:04:58.993347 20786 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:04:58.995121 20786 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:04:59.024207 20786 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ef165ef0d224817bb651983ada31be0: Generated new TSK 0
I20260806 03:04:59.025184 20786 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 03:04:59.042886 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 20701
I20260806 03:04:59.045101 20767 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 03:04:59.153167 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 20701
W20260806 03:04:59.176360 20685 connection.cc:535] client connection to 127.0.0.1:38839 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38839): Connection reset by peer (error 104)
2026-08-06T03:04:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2489 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785985105255475-32043-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 03:04:59 dist-test-slave-dhph krb5kdc[20805](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 03:04:59 dist-test-slave-dhph krb5kdc[20805](info): set up 2 sockets
Aug 06 03:04:59 dist-test-slave-dhph krb5kdc[20805](info): commencing operation
krb5kdc: starting...
W20260806 03:05:00.406296 32043 mini_kdc.cc:121] Time spent starting KDC: real 1.209s user 0.000s sys 0.056s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785985105255475-32043-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785985105255475-32043-0/krb5kdc/test-user.keytab.
Aug 06 03:05:00 dist-test-slave-dhph krb5kdc[20805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985500, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T03:05:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:05:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785985105255475-32043-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785985105255475-32043-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 03:05:00.788944 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785985105255475-32043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785985105255475-32043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785985105255475-32043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39247
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785985105255475-32043-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39247
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:05:01.185456 20820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:05:01.185825 20820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:05:01.186002 20820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:05:01.194770 20820 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:05:01.194999 20820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:05:01.195122 20820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:05:01.195272 20820 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:05:01.195374 20820 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:05:01.206386 20820 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785985105255475-32043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785985105255475-32043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39247
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785985105255475-32043-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785985105255475-32043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:05:01.207962 20820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:05:01.209405 20820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:05:01.225065 20825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:05:01.225095 20829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:05:01.225082 20826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:05:01.225947 20820 server_base.cc:1034] running on GCE node
I20260806 03:05:01.227087 20820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:05:01.228868 20820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:05:01.230180 20820 hybrid_clock.cc:648] HybridClock initialized: now 1785985501230143 us; error 53 us; skew 500 ppm
I20260806 03:05:01.230684 20820 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 03:05:01 dist-test-slave-dhph krb5kdc[20805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985501, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:05:01.240255 20820 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 03:05:01.242030 20820 webserver.cc:460] Webserver started at http://127.0.0.1:40587/ using document root <none> and password file <none>
I20260806 03:05:01.242834 20820 fs_manager.cc:359] Metadata directory not provided
I20260806 03:05:01.242990 20820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:05:01.243501 20820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:05:01.247038 20820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785985105255475-32043-0/minicluster-data/master-0/data/instance:
uuid: "8c6c40e24152466d8bf1e0f8f92a6696"
format_stamp: "Formatted at 2026-08-06 03:05:01 on dist-test-slave-dhph"
server_key: "177090266a12122a8cc3ace27a3b11ec"
server_key_iv: "836a40856357b2f285ed7545736df26e"
server_key_version: "encryptionkey@0"
I20260806 03:05:01.247905 20820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785985105255475-32043-0/minicluster-data/master-0/wal/instance:
uuid: "8c6c40e24152466d8bf1e0f8f92a6696"
format_stamp: "Formatted at 2026-08-06 03:05:01 on dist-test-slave-dhph"
server_key: "177090266a12122a8cc3ace27a3b11ec"
server_key_iv: "836a40856357b2f285ed7545736df26e"
server_key_version: "encryptionkey@0"
I20260806 03:05:01.254297 20820 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260806 03:05:01.258925 20836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:05:01.260586 20820 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 03:05:01.260833 20820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785985105255475-32043-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785985105255475-32043-0/minicluster-data/master-0/wal
uuid: "8c6c40e24152466d8bf1e0f8f92a6696"
format_stamp: "Formatted at 2026-08-06 03:05:01 on dist-test-slave-dhph"
server_key: "177090266a12122a8cc3ace27a3b11ec"
server_key_iv: "836a40856357b2f285ed7545736df26e"
server_key_version: "encryptionkey@0"
I20260806 03:05:01.261070 20820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785985105255475-32043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785985105255475-32043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785985105255475-32043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:05:01.281121 20820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:05:01.286175 20820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:05:01.286585 20820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:05:01.312680 20820 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39247
I20260806 03:05:01.312700 20889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39247 every 8 connection(s)
I20260806 03:05:01.314682 20820 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
I20260806 03:05:01.317670 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 20820
I20260806 03:05:01.317869 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785985105255475-32043-0/minicluster-data/master-0/wal/instance
I20260806 03:05:01.318450 32043 external_mini_cluster.cc:1428] Setting key 3d5aba0c40383800a6e986c850113bc6
I20260806 03:05:01.321535 20890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:05:01.334208 20893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:01.320541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55531 (local address 127.0.0.1:39247)
0806 03:05:01.322086 (+ 1545us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:01.322100 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:01.322145 (+ 45us) server_negotiation.cc:407] Connection header received
0806 03:05:01.323939 (+ 1794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:01.323964 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:01.324437 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:01.324916 (+ 479us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:01.325501 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:01.327214 (+ 1713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:01.328767 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:01.329583 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:01.329849 (+ 266us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:05:01.331491 (+ 1642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:05:01.331500 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:05:01.331520 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:05:01.332163 (+ 643us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:05:01.332252 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:05:01.332286 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:05:01.332514 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0806 03:05:01.332859 (+ 345us) server_negotiation.cc:299] Negotiation successful
0806 03:05:01.333059 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1313,"thread_start_us":1006,"threads_started":1}
I20260806 03:05:01.335664 20890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c6c40e24152466d8bf1e0f8f92a6696: Bootstrap starting.
I20260806 03:05:01.339071 32043 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 03:05:01.340135 20890 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c6c40e24152466d8bf1e0f8f92a6696: Neither blocks nor log segments found. Creating new log.
I20260806 03:05:01.341382 20890 log.cc:824] T 00000000000000000000000000000000 P 8c6c40e24152466d8bf1e0f8f92a6696: Log is configured to *not* fsync() on all Append() calls
I20260806 03:05:01.344990 20890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c6c40e24152466d8bf1e0f8f92a6696: No bootstrap required, opened a new log
I20260806 03:05:01.356328 20893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:01.345528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48185 (local address 127.0.0.1:39247)
0806 03:05:01.345798 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:01.345809 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:01.346574 (+ 765us) server_negotiation.cc:407] Connection header received
0806 03:05:01.346862 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:01.346866 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:01.346938 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:01.347036 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:01.348149 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:01.349329 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:01.351831 (+ 2502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:01.352427 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:01.352599 (+ 172us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:05:01.354188 (+ 1589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:05:01.354197 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:05:01.354201 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:05:01.354227 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:05:01.354279 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:05:01.354302 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:05:01.354416 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0806 03:05:01.354852 (+ 436us) server_negotiation.cc:299] Negotiation successful
0806 03:05:01.355015 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260806 03:05:01.359946 20890 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c6c40e24152466d8bf1e0f8f92a6696 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c6c40e24152466d8bf1e0f8f92a6696" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39247 } }
I20260806 03:05:01.360463 20890 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c6c40e24152466d8bf1e0f8f92a6696 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:05:01.360586 20890 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c6c40e24152466d8bf1e0f8f92a6696 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c6c40e24152466d8bf1e0f8f92a6696, State: Initialized, Role: FOLLOWER
I20260806 03:05:01.361217 20890 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c6c40e24152466d8bf1e0f8f92a6696 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c6c40e24152466d8bf1e0f8f92a6696" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39247 } }
I20260806 03:05:01.361486 20890 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c6c40e24152466d8bf1e0f8f92a6696 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:05:01.361642 20890 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c6c40e24152466d8bf1e0f8f92a6696 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:05:01.361845 20890 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c6c40e24152466d8bf1e0f8f92a6696 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:05:01.365320 20890 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c6c40e24152466d8bf1e0f8f92a6696 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c6c40e24152466d8bf1e0f8f92a6696" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39247 } }
I20260806 03:05:01.365856 20890 leader_election.cc:302] T 00000000000000000000000000000000 P 8c6c40e24152466d8bf1e0f8f92a6696 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c6c40e24152466d8bf1e0f8f92a6696; no voters:
I20260806 03:05:01.366813 20890 leader_election.cc:288] T 00000000000000000000000000000000 P 8c6c40e24152466d8bf1e0f8f92a6696 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:05:01.367066 20900 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c6c40e24152466d8bf1e0f8f92a6696 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:05:01.367947 20900 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c6c40e24152466d8bf1e0f8f92a6696 [term 1 LEADER]: Becoming Leader. State: Replica: 8c6c40e24152466d8bf1e0f8f92a6696, State: Running, Role: LEADER
I20260806 03:05:01.368810 20900 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c6c40e24152466d8bf1e0f8f92a6696 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c6c40e24152466d8bf1e0f8f92a6696" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39247 } }
I20260806 03:05:01.370456 20890 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c6c40e24152466d8bf1e0f8f92a6696 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:05:01.373219 20901 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c6c40e24152466d8bf1e0f8f92a6696 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c6c40e24152466d8bf1e0f8f92a6696" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c6c40e24152466d8bf1e0f8f92a6696" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39247 } } }
I20260806 03:05:01.373836 20901 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c6c40e24152466d8bf1e0f8f92a6696 [sys.catalog]: This master's current role is: LEADER
I20260806 03:05:01.374126 20902 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c6c40e24152466d8bf1e0f8f92a6696 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c6c40e24152466d8bf1e0f8f92a6696. Latest consensus state: current_term: 1 leader_uuid: "8c6c40e24152466d8bf1e0f8f92a6696" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c6c40e24152466d8bf1e0f8f92a6696" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39247 } } }
I20260806 03:05:01.374809 20902 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c6c40e24152466d8bf1e0f8f92a6696 [sys.catalog]: This master's current role is: LEADER
I20260806 03:05:01.379899 20905 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:05:01.387113 20905 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:05:01.399619 20905 catalog_manager.cc:1347] Generated new cluster ID: a778843ce90341a789d40e1f8b862ba3
I20260806 03:05:01.399777 20905 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:05:01.425345 20905 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:05:01.427198 20905 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:05:01.441602 20905 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c6c40e24152466d8bf1e0f8f92a6696: Generated new TSK 0
I20260806 03:05:01.443085 20905 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 03:05:01.461020 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 20820
W20260806 03:05:01.462857 20885 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:55531
W20260806 03:05:01.464557 32043 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 20820: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 03:05:01.464799 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 20820
W20260806 03:05:01.492295 20803 connection.cc:535] client connection to 127.0.0.1:39247 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39247): Connection reset by peer (error 104)
2026-08-06T03:05:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2315 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785985105255475-32043-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 03:05:01 dist-test-slave-dhph krb5kdc[20921](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 03:05:01 dist-test-slave-dhph krb5kdc[20921](info): set up 2 sockets
Aug 06 03:05:01 dist-test-slave-dhph krb5kdc[20921](info): commencing operation
krb5kdc: starting...
W20260806 03:05:02.815192 32043 mini_kdc.cc:121] Time spent starting KDC: real 1.301s user 0.000s sys 0.066s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785985105255475-32043-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785985105255475-32043-0/krb5kdc/test-user.keytab.
Aug 06 03:05:03 dist-test-slave-dhph krb5kdc[20921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T03:05:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:05:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785985105255475-32043-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785985105255475-32043-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 03:05:03.244114 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785985105255475-32043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785985105255475-32043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785985105255475-32043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36207
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785985105255475-32043-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36207
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:05:03.656411 20935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:05:03.656750 20935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:05:03.656876 20935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:05:03.665506 20935 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:05:03.665647 20935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:05:03.665804 20935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:05:03.665892 20935 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:05:03.665969 20935 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:05:03.676417 20935 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785985105255475-32043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785985105255475-32043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36207
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785985105255475-32043-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36207
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785985105255475-32043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:05:03.677883 20935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:05:03.679440 20935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:05:03.695449 20941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:05:03.695606 20944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:05:03.695374 20942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:05:03.696596 20935 server_base.cc:1034] running on GCE node
I20260806 03:05:03.697566 20935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:05:03.699237 20935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:05:03.700541 20935 hybrid_clock.cc:648] HybridClock initialized: now 1785985503700490 us; error 38 us; skew 500 ppm
I20260806 03:05:03.700975 20935 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 03:05:03 dist-test-slave-dhph krb5kdc[20921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985503, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:05:03.706465 20935 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 03:05:03.709012 20935 webserver.cc:460] Webserver started at http://127.0.0.1:36703/ using document root <none> and password file <none>
I20260806 03:05:03.709887 20935 fs_manager.cc:359] Metadata directory not provided
I20260806 03:05:03.710072 20935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:05:03.710541 20935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:05:03.715207 20935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785985105255475-32043-0/minicluster-data/master-0/data/instance:
uuid: "b66e4474fe80425eb0d5e44c73a50f43"
format_stamp: "Formatted at 2026-08-06 03:05:03 on dist-test-slave-dhph"
server_key: "3a8905dba428426308033d2496bd9387"
server_key_iv: "cfe10e975635280829bdc72680ad8be5"
server_key_version: "encryptionkey@0"
I20260806 03:05:03.716125 20935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785985105255475-32043-0/minicluster-data/master-0/wal/instance:
uuid: "b66e4474fe80425eb0d5e44c73a50f43"
format_stamp: "Formatted at 2026-08-06 03:05:03 on dist-test-slave-dhph"
server_key: "3a8905dba428426308033d2496bd9387"
server_key_iv: "cfe10e975635280829bdc72680ad8be5"
server_key_version: "encryptionkey@0"
I20260806 03:05:03.724891 20935 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.001s
I20260806 03:05:03.729661 20951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:05:03.731951 20935 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260806 03:05:03.732245 20935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785985105255475-32043-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785985105255475-32043-0/minicluster-data/master-0/wal
uuid: "b66e4474fe80425eb0d5e44c73a50f43"
format_stamp: "Formatted at 2026-08-06 03:05:03 on dist-test-slave-dhph"
server_key: "3a8905dba428426308033d2496bd9387"
server_key_iv: "cfe10e975635280829bdc72680ad8be5"
server_key_version: "encryptionkey@0"
I20260806 03:05:03.733210 20935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785985105255475-32043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785985105255475-32043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785985105255475-32043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:05:03.741657 20935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:05:03.748082 20935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:05:03.748608 20935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:05:03.774399 20935 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36207
I20260806 03:05:03.774504 21005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36207 every 8 connection(s)
I20260806 03:05:03.776204 20935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
I20260806 03:05:03.780817 21006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:05:03.781086 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 20935
I20260806 03:05:03.781284 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785985105255475-32043-0/minicluster-data/master-0/wal/instance
I20260806 03:05:03.781831 32043 external_mini_cluster.cc:1428] Setting key 10a32ff18e0268492229170ebc97b9ad
I20260806 03:05:03.793028 21009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:03.783842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45185 (local address 127.0.0.1:36207)
0806 03:05:03.784746 (+ 904us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:03.784762 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:03.784813 (+ 51us) server_negotiation.cc:407] Connection header received
0806 03:05:03.788249 (+ 3436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:03.788291 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:03.788820 (+ 529us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:03.789342 (+ 522us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:03.790982 (+ 1640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:05:03.790991 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:05:03.791008 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:05:03.791029 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:05:03.791129 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:05:03.791163 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:05:03.791419 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0806 03:05:03.791548 (+ 129us) server_negotiation.cc:299] Negotiation successful
0806 03:05:03.791746 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":368,"threads_started":1}
I20260806 03:05:03.797271 21006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b66e4474fe80425eb0d5e44c73a50f43: Bootstrap starting.
I20260806 03:05:03.798110 32043 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 03:05:03.800792 21006 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b66e4474fe80425eb0d5e44c73a50f43: Neither blocks nor log segments found. Creating new log.
I20260806 03:05:03.802125 21006 log.cc:824] T 00000000000000000000000000000000 P b66e4474fe80425eb0d5e44c73a50f43: Log is configured to *not* fsync() on all Append() calls
I20260806 03:05:03.807793 21006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b66e4474fe80425eb0d5e44c73a50f43: No bootstrap required, opened a new log
I20260806 03:05:03.808246 21009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:03.804213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51161 (local address 127.0.0.1:36207)
0806 03:05:03.804596 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:03.804607 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:03.804951 (+ 344us) server_negotiation.cc:407] Connection header received
0806 03:05:03.805191 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:03.805195 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:03.805434 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:03.805548 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:03.807205 (+ 1657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:05:03.807210 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:05:03.807215 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:05:03.807242 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:05:03.807285 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:05:03.807302 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:05:03.807451 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0806 03:05:03.807565 (+ 114us) server_negotiation.cc:299] Negotiation successful
0806 03:05:03.807640 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260806 03:05:03.818421 21006 raft_consensus.cc:348] T 00000000000000000000000000000000 P b66e4474fe80425eb0d5e44c73a50f43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b66e4474fe80425eb0d5e44c73a50f43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36207 } }
I20260806 03:05:03.819126 21006 raft_consensus.cc:374] T 00000000000000000000000000000000 P b66e4474fe80425eb0d5e44c73a50f43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:05:03.819265 21006 raft_consensus.cc:729] T 00000000000000000000000000000000 P b66e4474fe80425eb0d5e44c73a50f43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b66e4474fe80425eb0d5e44c73a50f43, State: Initialized, Role: FOLLOWER
I20260806 03:05:03.820430 21006 consensus_queue.cc:260] T 00000000000000000000000000000000 P b66e4474fe80425eb0d5e44c73a50f43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b66e4474fe80425eb0d5e44c73a50f43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36207 } }
I20260806 03:05:03.820791 21006 raft_consensus.cc:388] T 00000000000000000000000000000000 P b66e4474fe80425eb0d5e44c73a50f43 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:05:03.820977 21006 raft_consensus.cc:482] T 00000000000000000000000000000000 P b66e4474fe80425eb0d5e44c73a50f43 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:05:03.821189 21006 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b66e4474fe80425eb0d5e44c73a50f43 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:05:03.824102 21006 raft_consensus.cc:504] T 00000000000000000000000000000000 P b66e4474fe80425eb0d5e44c73a50f43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b66e4474fe80425eb0d5e44c73a50f43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36207 } }
I20260806 03:05:03.824790 21006 leader_election.cc:302] T 00000000000000000000000000000000 P b66e4474fe80425eb0d5e44c73a50f43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b66e4474fe80425eb0d5e44c73a50f43; no voters:
I20260806 03:05:03.825518 21006 leader_election.cc:288] T 00000000000000000000000000000000 P b66e4474fe80425eb0d5e44c73a50f43 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:05:03.825721 21016 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b66e4474fe80425eb0d5e44c73a50f43 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:05:03.827421 21016 raft_consensus.cc:686] T 00000000000000000000000000000000 P b66e4474fe80425eb0d5e44c73a50f43 [term 1 LEADER]: Becoming Leader. State: Replica: b66e4474fe80425eb0d5e44c73a50f43, State: Running, Role: LEADER
I20260806 03:05:03.828284 21006 sys_catalog.cc:564] T 00000000000000000000000000000000 P b66e4474fe80425eb0d5e44c73a50f43 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:05:03.828152 21016 consensus_queue.cc:237] T 00000000000000000000000000000000 P b66e4474fe80425eb0d5e44c73a50f43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b66e4474fe80425eb0d5e44c73a50f43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36207 } }
I20260806 03:05:03.832453 21018 sys_catalog.cc:455] T 00000000000000000000000000000000 P b66e4474fe80425eb0d5e44c73a50f43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b66e4474fe80425eb0d5e44c73a50f43. Latest consensus state: current_term: 1 leader_uuid: "b66e4474fe80425eb0d5e44c73a50f43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b66e4474fe80425eb0d5e44c73a50f43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36207 } } }
I20260806 03:05:03.833016 21018 sys_catalog.cc:458] T 00000000000000000000000000000000 P b66e4474fe80425eb0d5e44c73a50f43 [sys.catalog]: This master's current role is: LEADER
I20260806 03:05:03.834076 21017 sys_catalog.cc:455] T 00000000000000000000000000000000 P b66e4474fe80425eb0d5e44c73a50f43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b66e4474fe80425eb0d5e44c73a50f43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b66e4474fe80425eb0d5e44c73a50f43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36207 } } }
I20260806 03:05:03.834605 21017 sys_catalog.cc:458] T 00000000000000000000000000000000 P b66e4474fe80425eb0d5e44c73a50f43 [sys.catalog]: This master's current role is: LEADER
I20260806 03:05:03.839795 21024 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:05:03.846972 21024 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:05:03.858269 21024 catalog_manager.cc:1347] Generated new cluster ID: d4aa6f1cb3dc4080b9714b30a4732102
I20260806 03:05:03.858451 21024 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:05:03.909034 21024 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:05:03.909590 21024 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:05:03.916461 21024 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b66e4474fe80425eb0d5e44c73a50f43: Generated new TSK 0
I20260806 03:05:03.917455 21024 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 03:05:03.922559 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 20935
W20260806 03:05:03.924263 21001 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:45185
W20260806 03:05:03.925494 32043 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 20935: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 03:05:03.925869 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 20935
2026-08-06T03:05:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (2463 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785985105255475-32043-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 03:05:04 dist-test-slave-dhph krb5kdc[21038](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 03:05:04 dist-test-slave-dhph krb5kdc[21038](info): set up 2 sockets
Aug 06 03:05:04 dist-test-slave-dhph krb5kdc[21038](info): commencing operation
krb5kdc: starting...
W20260806 03:05:05.252764 32043 mini_kdc.cc:121] Time spent starting KDC: real 1.276s user 0.000s sys 0.062s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785985105255475-32043-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785985105255475-32043-0/krb5kdc/test-user.keytab.
Aug 06 03:05:05 dist-test-slave-dhph krb5kdc[21038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985505, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T03:05:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:05:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785985105255475-32043-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785985105255475-32043-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 03:05:05.656298 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785985105255475-32043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785985105255475-32043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785985105255475-32043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40663
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785985105255475-32043-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40663
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:05:06.046053 21052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:05:06.046392 21052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:05:06.046620 21052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:05:06.054325 21052 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:05:06.054451 21052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:05:06.054548 21052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:05:06.054690 21052 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:05:06.054764 21052 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:05:06.065697 21052 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785985105255475-32043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785985105255475-32043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40663
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785985105255475-32043-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785985105255475-32043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:05:06.067268 21052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:05:06.068600 21052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:05:06.081920 21057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:05:06.081945 21060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:05:06.081882 21058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:05:06.083629 21052 server_base.cc:1034] running on GCE node
I20260806 03:05:06.084424 21052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:05:06.085902 21052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:05:06.087178 21052 hybrid_clock.cc:648] HybridClock initialized: now 1785985506087126 us; error 38 us; skew 500 ppm
I20260806 03:05:06.087616 21052 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 03:05:06 dist-test-slave-dhph krb5kdc[21038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985506, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:05:06.091122 21052 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 03:05:06.092808 21052 webserver.cc:460] Webserver started at http://127.0.0.1:44307/ using document root <none> and password file <none>
I20260806 03:05:06.093510 21052 fs_manager.cc:359] Metadata directory not provided
I20260806 03:05:06.093673 21052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:05:06.093986 21052 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:05:06.097242 21052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785985105255475-32043-0/minicluster-data/master-0/data/instance:
uuid: "41f16b4400ed4ee9894855a372cdb756"
format_stamp: "Formatted at 2026-08-06 03:05:06 on dist-test-slave-dhph"
server_key: "4518b1b817798c3d693978848ab32a4f"
server_key_iv: "dfa734ac84a664e68cc5e972700fd71b"
server_key_version: "encryptionkey@0"
I20260806 03:05:06.098093 21052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785985105255475-32043-0/minicluster-data/master-0/wal/instance:
uuid: "41f16b4400ed4ee9894855a372cdb756"
format_stamp: "Formatted at 2026-08-06 03:05:06 on dist-test-slave-dhph"
server_key: "4518b1b817798c3d693978848ab32a4f"
server_key_iv: "dfa734ac84a664e68cc5e972700fd71b"
server_key_version: "encryptionkey@0"
I20260806 03:05:06.103987 21052 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260806 03:05:06.108568 21068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:05:06.110416 21052 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260806 03:05:06.110785 21052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785985105255475-32043-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785985105255475-32043-0/minicluster-data/master-0/wal
uuid: "41f16b4400ed4ee9894855a372cdb756"
format_stamp: "Formatted at 2026-08-06 03:05:06 on dist-test-slave-dhph"
server_key: "4518b1b817798c3d693978848ab32a4f"
server_key_iv: "dfa734ac84a664e68cc5e972700fd71b"
server_key_version: "encryptionkey@0"
I20260806 03:05:06.111030 21052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785985105255475-32043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785985105255475-32043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785985105255475-32043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:05:06.129257 21052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:05:06.135533 21052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:05:06.135890 21052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:05:06.161857 21052 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40663
I20260806 03:05:06.161883 21121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40663 every 8 connection(s)
I20260806 03:05:06.163879 21052 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
I20260806 03:05:06.167111 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 21052
I20260806 03:05:06.167310 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785985105255475-32043-0/minicluster-data/master-0/wal/instance
I20260806 03:05:06.167809 32043 external_mini_cluster.cc:1428] Setting key 6f329b923d53a617431352aea0990065
I20260806 03:05:06.168419 21122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:05:06.185298 21122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41f16b4400ed4ee9894855a372cdb756: Bootstrap starting.
Aug 06 03:05:06 dist-test-slave-dhph krb5kdc[21038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985505, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 03:05:06.190526 21122 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41f16b4400ed4ee9894855a372cdb756: Neither blocks nor log segments found. Creating new log.
I20260806 03:05:06.192207 21122 log.cc:824] T 00000000000000000000000000000000 P 41f16b4400ed4ee9894855a372cdb756: Log is configured to *not* fsync() on all Append() calls
I20260806 03:05:06.198097 21122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41f16b4400ed4ee9894855a372cdb756: No bootstrap required, opened a new log
I20260806 03:05:06.199671 21125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:06.170216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38873 (local address 127.0.0.1:40663)
0806 03:05:06.171231 (+ 1015us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:06.171248 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:06.171305 (+ 57us) server_negotiation.cc:407] Connection header received
0806 03:05:06.173989 (+ 2684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:06.174023 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:06.174771 (+ 748us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:06.175336 (+ 565us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:06.176982 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:06.179261 (+ 2279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:06.180841 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:06.182132 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:06.182374 (+ 242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:05:06.187790 (+ 5416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:05:06.187804 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:05:06.187827 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:05:06.187920 (+ 93us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:05:06.192605 (+ 4685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:05:06.193840 (+ 1235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:05:06.193852 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:05:06.193859 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:05:06.194125 (+ 266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:05:06.194655 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:05:06.194660 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:05:06.194664 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:05:06.195471 (+ 807us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:05:06.195807 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0806 03:05:06.197979 (+ 2172us) server_negotiation.cc:299] Negotiation successful
0806 03:05:06.198489 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":726,"thread_start_us":455,"threads_started":1}
I20260806 03:05:06.203920 32043 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 03:05:06.207398 21122 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41f16b4400ed4ee9894855a372cdb756 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41f16b4400ed4ee9894855a372cdb756" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40663 } }
I20260806 03:05:06.207955 21122 raft_consensus.cc:374] T 00000000000000000000000000000000 P 41f16b4400ed4ee9894855a372cdb756 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:05:06.208065 21122 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41f16b4400ed4ee9894855a372cdb756 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41f16b4400ed4ee9894855a372cdb756, State: Initialized, Role: FOLLOWER
I20260806 03:05:06.208767 21122 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41f16b4400ed4ee9894855a372cdb756 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41f16b4400ed4ee9894855a372cdb756" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40663 } }
I20260806 03:05:06.209054 21122 raft_consensus.cc:388] T 00000000000000000000000000000000 P 41f16b4400ed4ee9894855a372cdb756 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:05:06.209213 21122 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41f16b4400ed4ee9894855a372cdb756 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:05:06.209416 21122 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41f16b4400ed4ee9894855a372cdb756 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:05:06.213070 21122 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41f16b4400ed4ee9894855a372cdb756 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41f16b4400ed4ee9894855a372cdb756" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40663 } }
I20260806 03:05:06.213829 21122 leader_election.cc:302] T 00000000000000000000000000000000 P 41f16b4400ed4ee9894855a372cdb756 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 41f16b4400ed4ee9894855a372cdb756; no voters:
I20260806 03:05:06.214564 21122 leader_election.cc:288] T 00000000000000000000000000000000 P 41f16b4400ed4ee9894855a372cdb756 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:05:06.214749 21132 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41f16b4400ed4ee9894855a372cdb756 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:05:06.217672 21122 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41f16b4400ed4ee9894855a372cdb756 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:05:06.220891 21132 raft_consensus.cc:686] T 00000000000000000000000000000000 P 41f16b4400ed4ee9894855a372cdb756 [term 1 LEADER]: Becoming Leader. State: Replica: 41f16b4400ed4ee9894855a372cdb756, State: Running, Role: LEADER
I20260806 03:05:06.221853 21132 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41f16b4400ed4ee9894855a372cdb756 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41f16b4400ed4ee9894855a372cdb756" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40663 } }
I20260806 03:05:06.229912 21125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:06.210419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52657 (local address 127.0.0.1:40663)
0806 03:05:06.210756 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:06.210766 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:06.211230 (+ 464us) server_negotiation.cc:407] Connection header received
0806 03:05:06.211424 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:06.211429 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:06.211507 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:06.211621 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:06.213794 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:06.214959 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:06.216961 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:06.217562 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:06.217730 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:05:06.220697 (+ 2967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:05:06.220708 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:05:06.220714 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:05:06.220769 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:05:06.225335 (+ 4566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:05:06.226521 (+ 1186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:05:06.226531 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:05:06.226537 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:05:06.226759 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:05:06.227413 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:05:06.227421 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:05:06.227426 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:05:06.228012 (+ 586us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:05:06.228375 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0806 03:05:06.229113 (+ 738us) server_negotiation.cc:299] Negotiation successful
0806 03:05:06.229424 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260806 03:05:06.232357 21134 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41f16b4400ed4ee9894855a372cdb756 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41f16b4400ed4ee9894855a372cdb756. Latest consensus state: current_term: 1 leader_uuid: "41f16b4400ed4ee9894855a372cdb756" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41f16b4400ed4ee9894855a372cdb756" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40663 } } }
I20260806 03:05:06.233093 21134 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41f16b4400ed4ee9894855a372cdb756 [sys.catalog]: This master's current role is: LEADER
I20260806 03:05:06.233093 21133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41f16b4400ed4ee9894855a372cdb756 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "41f16b4400ed4ee9894855a372cdb756" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41f16b4400ed4ee9894855a372cdb756" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40663 } } }
I20260806 03:05:06.234010 21133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41f16b4400ed4ee9894855a372cdb756 [sys.catalog]: This master's current role is: LEADER
W20260806 03:05:06.239372 21148 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 41f16b4400ed4ee9894855a372cdb756: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 03:05:06.239543 21148 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 03:05:06.239696 21147 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:05:06.241544 21147 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:05:06.252180 21147 catalog_manager.cc:1347] Generated new cluster ID: fc61850b8d89416399b68417f7c9ca9f
I20260806 03:05:06.252494 21147 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:05:06.262001 21147 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:05:06.263501 21147 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:05:06.289434 21147 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 41f16b4400ed4ee9894855a372cdb756: Generated new TSK 0
I20260806 03:05:06.290673 21147 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 03:05:06.308758 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 21052
I20260806 03:05:06.310320 21116 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 03:05:06.414537 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 21052
2026-08-06T03:05:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2481 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785985105255475-32043-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 03:05:06 dist-test-slave-dhph krb5kdc[21154](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 03:05:06 dist-test-slave-dhph krb5kdc[21154](info): set up 2 sockets
Aug 06 03:05:06 dist-test-slave-dhph krb5kdc[21154](info): commencing operation
krb5kdc: starting...
W20260806 03:05:07.663055 32043 mini_kdc.cc:121] Time spent starting KDC: real 1.205s user 0.001s sys 0.056s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785985105255475-32043-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785985105255475-32043-0/krb5kdc/test-user.keytab.
Aug 06 03:05:07 dist-test-slave-dhph krb5kdc[21154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985507, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T03:05:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:05:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785985105255475-32043-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785985105255475-32043-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 03:05:08.100430 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785985105255475-32043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785985105255475-32043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785985105255475-32043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39483
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785985105255475-32043-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39483
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:05:08.506373 21167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:05:08.506763 21167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:05:08.506880 21167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:05:08.515798 21167 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:05:08.515949 21167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:05:08.516003 21167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:05:08.516072 21167 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:05:08.516124 21167 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:05:08.527515 21167 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785985105255475-32043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785985105255475-32043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39483
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785985105255475-32043-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785985105255475-32043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:05:08.528872 21167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:05:08.530170 21167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:05:08.543882 21173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:05:08.544222 21174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:05:08.544396 21176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:05:08.545172 21167 server_base.cc:1034] running on GCE node
I20260806 03:05:08.546517 21167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:05:08.548854 21167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:05:08.552407 21167 hybrid_clock.cc:648] HybridClock initialized: now 1785985508552282 us; error 114 us; skew 500 ppm
I20260806 03:05:08.553030 21167 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 03:05:08 dist-test-slave-dhph krb5kdc[21154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985508, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:05:08.556879 21167 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 03:05:08.559111 21167 webserver.cc:460] Webserver started at http://127.0.0.1:39845/ using document root <none> and password file <none>
I20260806 03:05:08.560111 21167 fs_manager.cc:359] Metadata directory not provided
I20260806 03:05:08.560333 21167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:05:08.560691 21167 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:05:08.564277 21167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785985105255475-32043-0/minicluster-data/master-0/data/instance:
uuid: "93b6ffb7c566489ab6d06b47553da2df"
format_stamp: "Formatted at 2026-08-06 03:05:08 on dist-test-slave-dhph"
server_key: "5990d5f217a33fe24d83305420b4835f"
server_key_iv: "3f8e178188526bd718971ccc65bb4876"
server_key_version: "encryptionkey@0"
I20260806 03:05:08.565130 21167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785985105255475-32043-0/minicluster-data/master-0/wal/instance:
uuid: "93b6ffb7c566489ab6d06b47553da2df"
format_stamp: "Formatted at 2026-08-06 03:05:08 on dist-test-slave-dhph"
server_key: "5990d5f217a33fe24d83305420b4835f"
server_key_iv: "3f8e178188526bd718971ccc65bb4876"
server_key_version: "encryptionkey@0"
I20260806 03:05:08.572194 21167 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20260806 03:05:08.576432 21183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:05:08.578316 21167 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 03:05:08.578685 21167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785985105255475-32043-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785985105255475-32043-0/minicluster-data/master-0/wal
uuid: "93b6ffb7c566489ab6d06b47553da2df"
format_stamp: "Formatted at 2026-08-06 03:05:08 on dist-test-slave-dhph"
server_key: "5990d5f217a33fe24d83305420b4835f"
server_key_iv: "3f8e178188526bd718971ccc65bb4876"
server_key_version: "encryptionkey@0"
I20260806 03:05:08.578922 21167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785985105255475-32043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785985105255475-32043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785985105255475-32043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:05:08.606354 21167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:05:08.612322 21167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:05:08.612697 21167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:05:08.636668 21167 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39483
I20260806 03:05:08.636718 21235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39483 every 8 connection(s)
I20260806 03:05:08.638159 21167 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
I20260806 03:05:08.642318 21237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:05:08.644452 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 21167
I20260806 03:05:08.644673 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785985105255475-32043-0/minicluster-data/master-0/wal/instance
I20260806 03:05:08.645262 32043 external_mini_cluster.cc:1428] Setting key 73baffd83d8915c867a91a7e0a9ea975
I20260806 03:05:08.657814 21237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93b6ffb7c566489ab6d06b47553da2df: Bootstrap starting.
I20260806 03:05:08.661702 21237 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93b6ffb7c566489ab6d06b47553da2df: Neither blocks nor log segments found. Creating new log.
I20260806 03:05:08.662011 21240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:08.647475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54995 (local address 127.0.0.1:39483)
0806 03:05:08.648582 (+ 1107us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:08.648597 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:08.648661 (+ 64us) server_negotiation.cc:407] Connection header received
0806 03:05:08.651110 (+ 2449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:08.651140 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:08.651945 (+ 805us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:08.652480 (+ 535us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:08.653100 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:08.655508 (+ 2408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:08.657231 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:08.657965 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:08.658182 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:05:08.659515 (+ 1333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:05:08.659526 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:05:08.659541 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:05:08.660144 (+ 603us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:05:08.660237 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:05:08.660264 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:05:08.660403 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0806 03:05:08.660626 (+ 223us) server_negotiation.cc:299] Negotiation successful
0806 03:05:08.660921 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":836,"thread_start_us":474,"threads_started":1}
I20260806 03:05:08.663507 21237 log.cc:824] T 00000000000000000000000000000000 P 93b6ffb7c566489ab6d06b47553da2df: Log is configured to *not* fsync() on all Append() calls
I20260806 03:05:08.666162 32043 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 03:05:08.667532 21237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93b6ffb7c566489ab6d06b47553da2df: No bootstrap required, opened a new log
I20260806 03:05:08.677290 21237 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93b6ffb7c566489ab6d06b47553da2df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93b6ffb7c566489ab6d06b47553da2df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39483 } }
I20260806 03:05:08.677872 21237 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93b6ffb7c566489ab6d06b47553da2df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:05:08.678016 21237 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93b6ffb7c566489ab6d06b47553da2df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93b6ffb7c566489ab6d06b47553da2df, State: Initialized, Role: FOLLOWER
I20260806 03:05:08.678892 21237 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93b6ffb7c566489ab6d06b47553da2df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93b6ffb7c566489ab6d06b47553da2df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39483 } }
I20260806 03:05:08.679253 21237 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93b6ffb7c566489ab6d06b47553da2df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:05:08.679663 21240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:08.671683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42407 (local address 127.0.0.1:39483)
0806 03:05:08.671911 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:08.671923 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:08.672328 (+ 405us) server_negotiation.cc:407] Connection header received
0806 03:05:08.672468 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:08.672473 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:08.672565 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:08.672687 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:08.673455 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:08.674708 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:08.676310 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:08.676998 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:08.677149 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:05:08.678366 (+ 1217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:05:08.678373 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:05:08.678378 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:05:08.678403 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:05:08.678454 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:05:08.678481 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:05:08.678609 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0806 03:05:08.678844 (+ 235us) server_negotiation.cc:299] Negotiation successful
0806 03:05:08.678968 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260806 03:05:08.679800 21237 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93b6ffb7c566489ab6d06b47553da2df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:05:08.680037 21237 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93b6ffb7c566489ab6d06b47553da2df [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:05:08.682827 21237 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93b6ffb7c566489ab6d06b47553da2df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93b6ffb7c566489ab6d06b47553da2df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39483 } }
I20260806 03:05:08.683406 21237 leader_election.cc:302] T 00000000000000000000000000000000 P 93b6ffb7c566489ab6d06b47553da2df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93b6ffb7c566489ab6d06b47553da2df; no voters:
I20260806 03:05:08.683975 21237 leader_election.cc:288] T 00000000000000000000000000000000 P 93b6ffb7c566489ab6d06b47553da2df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:05:08.684173 21247 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93b6ffb7c566489ab6d06b47553da2df [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:05:08.684911 21247 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93b6ffb7c566489ab6d06b47553da2df [term 1 LEADER]: Becoming Leader. State: Replica: 93b6ffb7c566489ab6d06b47553da2df, State: Running, Role: LEADER
I20260806 03:05:08.685549 21247 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93b6ffb7c566489ab6d06b47553da2df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93b6ffb7c566489ab6d06b47553da2df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39483 } }
I20260806 03:05:08.686357 21237 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93b6ffb7c566489ab6d06b47553da2df [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:05:08.688920 21249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93b6ffb7c566489ab6d06b47553da2df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93b6ffb7c566489ab6d06b47553da2df. Latest consensus state: current_term: 1 leader_uuid: "93b6ffb7c566489ab6d06b47553da2df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93b6ffb7c566489ab6d06b47553da2df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39483 } } }
I20260806 03:05:08.689411 21249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93b6ffb7c566489ab6d06b47553da2df [sys.catalog]: This master's current role is: LEADER
I20260806 03:05:08.690781 21256 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:05:08.696048 21256 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:05:08.689931 21248 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93b6ffb7c566489ab6d06b47553da2df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "93b6ffb7c566489ab6d06b47553da2df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93b6ffb7c566489ab6d06b47553da2df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39483 } } }
I20260806 03:05:08.698699 21248 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93b6ffb7c566489ab6d06b47553da2df [sys.catalog]: This master's current role is: LEADER
I20260806 03:05:08.707623 21256 catalog_manager.cc:1347] Generated new cluster ID: c18499a7a7a34d3781c98fde034fb275
I20260806 03:05:08.707762 21256 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:05:08.737564 21256 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:05:08.739573 21256 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:05:08.753137 21256 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 93b6ffb7c566489ab6d06b47553da2df: Generated new TSK 0
I20260806 03:05:08.754333 21256 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 03:05:08.764961 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 21167
W20260806 03:05:08.766405 21231 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:54995
W20260806 03:05:08.767385 32043 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 21167: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 03:05:08.767596 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 21167
2026-08-06T03:05:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2354 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785985105255475-32043-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 03:05:08 dist-test-slave-dhph krb5kdc[21270](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 03:05:08 dist-test-slave-dhph krb5kdc[21270](info): set up 2 sockets
Aug 06 03:05:08 dist-test-slave-dhph krb5kdc[21270](info): commencing operation
krb5kdc: starting...
W20260806 03:05:10.017915 32043 mini_kdc.cc:121] Time spent starting KDC: real 1.205s user 0.000s sys 0.060s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785985105255475-32043-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785985105255475-32043-0/krb5kdc/test-user.keytab.
Aug 06 03:05:10 dist-test-slave-dhph krb5kdc[21270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T03:05:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:05:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785985105255475-32043-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785985105255475-32043-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 03:05:10.435246 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785985105255475-32043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785985105255475-32043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785985105255475-32043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36955
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785985105255475-32043-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36955
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:05:10.826210 21284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:05:10.826570 21284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:05:10.826797 21284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:05:10.834383 21284 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:05:10.834520 21284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:05:10.834645 21284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:05:10.834733 21284 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:05:10.834800 21284 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:05:10.846426 21284 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785985105255475-32043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785985105255475-32043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36955
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785985105255475-32043-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785985105255475-32043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:05:10.848738 21284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:05:10.850373 21284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:05:10.864086 21290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:05:10.864681 21292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:05:10.864774 21289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:05:10.866729 21284 server_base.cc:1034] running on GCE node
I20260806 03:05:10.867715 21284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:05:10.869395 21284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:05:10.870746 21284 hybrid_clock.cc:648] HybridClock initialized: now 1785985510870681 us; error 57 us; skew 500 ppm
I20260806 03:05:10.871182 21284 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 03:05:10 dist-test-slave-dhph krb5kdc[21270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985510, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:05:10.875026 21284 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 03:05:10.876765 21284 webserver.cc:460] Webserver started at http://127.0.0.1:43761/ using document root <none> and password file <none>
I20260806 03:05:10.877473 21284 fs_manager.cc:359] Metadata directory not provided
I20260806 03:05:10.877599 21284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:05:10.877871 21284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:05:10.881210 21284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785985105255475-32043-0/minicluster-data/master-0/data/instance:
uuid: "16b7d947b7174346a126712eba5db12a"
format_stamp: "Formatted at 2026-08-06 03:05:10 on dist-test-slave-dhph"
server_key: "ad11eaf64fba30c68e77e05b9cbf038a"
server_key_iv: "b9b95d89d90e0d4504a55763b7005259"
server_key_version: "encryptionkey@0"
I20260806 03:05:10.882309 21284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785985105255475-32043-0/minicluster-data/master-0/wal/instance:
uuid: "16b7d947b7174346a126712eba5db12a"
format_stamp: "Formatted at 2026-08-06 03:05:10 on dist-test-slave-dhph"
server_key: "ad11eaf64fba30c68e77e05b9cbf038a"
server_key_iv: "b9b95d89d90e0d4504a55763b7005259"
server_key_version: "encryptionkey@0"
I20260806 03:05:10.888459 21284 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.003s
I20260806 03:05:10.892835 21299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:05:10.894537 21284 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 03:05:10.894829 21284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785985105255475-32043-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785985105255475-32043-0/minicluster-data/master-0/wal
uuid: "16b7d947b7174346a126712eba5db12a"
format_stamp: "Formatted at 2026-08-06 03:05:10 on dist-test-slave-dhph"
server_key: "ad11eaf64fba30c68e77e05b9cbf038a"
server_key_iv: "b9b95d89d90e0d4504a55763b7005259"
server_key_version: "encryptionkey@0"
I20260806 03:05:10.895056 21284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785985105255475-32043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785985105255475-32043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785985105255475-32043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:05:10.901880 21284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:05:10.907532 21284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:05:10.907841 21284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:05:10.932185 21284 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36955
I20260806 03:05:10.932235 21350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36955 every 8 connection(s)
I20260806 03:05:10.933701 21284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
I20260806 03:05:10.938736 21351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:05:10.940132 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 21284
I20260806 03:05:10.940363 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785985105255475-32043-0/minicluster-data/master-0/wal/instance
I20260806 03:05:10.940949 32043 external_mini_cluster.cc:1428] Setting key 873bc0dc65901aeca45dca71b69529a0
I20260806 03:05:10.951922 21354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:10.943081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37655 (local address 127.0.0.1:36955)
0806 03:05:10.944456 (+ 1375us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:10.944470 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:10.944613 (+ 143us) server_negotiation.cc:407] Connection header received
0806 03:05:10.947243 (+ 2630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:10.947274 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:10.947952 (+ 678us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:10.948574 (+ 622us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:10.949891 (+ 1317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:05:10.949899 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:05:10.949913 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:05:10.949931 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:05:10.950000 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:05:10.950023 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:05:10.950493 (+ 470us) server_negotiation.cc:1035] Waiting for connection context
0806 03:05:10.950680 (+ 187us) server_negotiation.cc:299] Negotiation successful
0806 03:05:10.950889 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":974,"thread_start_us":676,"threads_started":1}
I20260806 03:05:10.956733 32043 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 03:05:10.957710 21351 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16b7d947b7174346a126712eba5db12a: Bootstrap starting.
I20260806 03:05:10.961620 21351 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16b7d947b7174346a126712eba5db12a: Neither blocks nor log segments found. Creating new log.
I20260806 03:05:10.963233 21351 log.cc:824] T 00000000000000000000000000000000 P 16b7d947b7174346a126712eba5db12a: Log is configured to *not* fsync() on all Append() calls
I20260806 03:05:10.965776 21354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:10.962031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41665 (local address 127.0.0.1:36955)
0806 03:05:10.962257 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:10.962265 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:10.962609 (+ 344us) server_negotiation.cc:407] Connection header received
0806 03:05:10.962844 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:10.962851 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:10.962925 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:10.963070 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:10.964985 (+ 1915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:05:10.964989 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:05:10.964993 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:05:10.965010 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:05:10.965053 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:05:10.965070 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:05:10.965237 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0806 03:05:10.965428 (+ 191us) server_negotiation.cc:299] Negotiation successful
0806 03:05:10.965493 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260806 03:05:10.967581 21351 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16b7d947b7174346a126712eba5db12a: No bootstrap required, opened a new log
I20260806 03:05:10.975406 21351 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16b7d947b7174346a126712eba5db12a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16b7d947b7174346a126712eba5db12a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36955 } }
I20260806 03:05:10.976130 21351 raft_consensus.cc:374] T 00000000000000000000000000000000 P 16b7d947b7174346a126712eba5db12a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:05:10.976292 21351 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16b7d947b7174346a126712eba5db12a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16b7d947b7174346a126712eba5db12a, State: Initialized, Role: FOLLOWER
I20260806 03:05:10.977049 21351 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16b7d947b7174346a126712eba5db12a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16b7d947b7174346a126712eba5db12a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36955 } }
I20260806 03:05:10.977347 21351 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16b7d947b7174346a126712eba5db12a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:05:10.977556 21351 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16b7d947b7174346a126712eba5db12a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:05:10.977775 21351 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16b7d947b7174346a126712eba5db12a [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:05:10.980921 21351 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16b7d947b7174346a126712eba5db12a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16b7d947b7174346a126712eba5db12a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36955 } }
I20260806 03:05:10.981606 21351 leader_election.cc:302] T 00000000000000000000000000000000 P 16b7d947b7174346a126712eba5db12a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16b7d947b7174346a126712eba5db12a; no voters:
I20260806 03:05:10.982321 21351 leader_election.cc:288] T 00000000000000000000000000000000 P 16b7d947b7174346a126712eba5db12a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:05:10.982481 21361 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16b7d947b7174346a126712eba5db12a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:05:10.983950 21361 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16b7d947b7174346a126712eba5db12a [term 1 LEADER]: Becoming Leader. State: Replica: 16b7d947b7174346a126712eba5db12a, State: Running, Role: LEADER
I20260806 03:05:10.984648 21361 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16b7d947b7174346a126712eba5db12a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16b7d947b7174346a126712eba5db12a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36955 } }
I20260806 03:05:10.985133 21351 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16b7d947b7174346a126712eba5db12a [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:05:10.988147 21362 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16b7d947b7174346a126712eba5db12a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "16b7d947b7174346a126712eba5db12a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16b7d947b7174346a126712eba5db12a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36955 } } }
I20260806 03:05:10.988679 21362 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16b7d947b7174346a126712eba5db12a [sys.catalog]: This master's current role is: LEADER
I20260806 03:05:10.989547 21363 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16b7d947b7174346a126712eba5db12a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16b7d947b7174346a126712eba5db12a. Latest consensus state: current_term: 1 leader_uuid: "16b7d947b7174346a126712eba5db12a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16b7d947b7174346a126712eba5db12a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36955 } } }
I20260806 03:05:10.990078 21363 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16b7d947b7174346a126712eba5db12a [sys.catalog]: This master's current role is: LEADER
I20260806 03:05:10.991858 21371 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:05:10.997409 21371 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:05:11.008683 21371 catalog_manager.cc:1347] Generated new cluster ID: 1712cc9e276c4ad99d917d1cfafbdaa7
I20260806 03:05:11.008872 21371 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:05:11.023576 21371 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:05:11.024346 21371 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:05:11.031559 21371 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 16b7d947b7174346a126712eba5db12a: Generated new TSK 0
I20260806 03:05:11.032526 21371 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 03:05:11.045219 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 21284
W20260806 03:05:11.046476 21346 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:37655
W20260806 03:05:11.047472 32043 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 21284: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 03:05:11.047605 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 21284
2026-08-06T03:05:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (2278 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785985105255475-32043-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 03:05:11 dist-test-slave-dhph krb5kdc[21383](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 03:05:11 dist-test-slave-dhph krb5kdc[21383](info): set up 2 sockets
Aug 06 03:05:11 dist-test-slave-dhph krb5kdc[21383](info): commencing operation
krb5kdc: starting...
W20260806 03:05:12.325598 32043 mini_kdc.cc:121] Time spent starting KDC: real 1.235s user 0.000s sys 0.058s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785985105255475-32043-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785985105255475-32043-0/krb5kdc/test-user.keytab.
Aug 06 03:05:12 dist-test-slave-dhph krb5kdc[21383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985512, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T03:05:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:05:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785985105255475-32043-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785985105255475-32043-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 03:05:12.727778 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785985105255475-32043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785985105255475-32043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785985105255475-32043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44581
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785985105255475-32043-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44581
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:05:13.151167 21397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:05:13.151517 21397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:05:13.151929 21397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:05:13.161767 21397 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:05:13.161919 21397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:05:13.162022 21397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:05:13.162425 21397 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:05:13.162756 21397 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:05:13.175096 21397 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785985105255475-32043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785985105255475-32043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44581
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785985105255475-32043-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785985105255475-32043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:05:13.176556 21397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:05:13.177984 21397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:05:13.193629 21406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:05:13.193715 21402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:05:13.193653 21403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:05:13.195282 21397 server_base.cc:1034] running on GCE node
I20260806 03:05:13.196626 21397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:05:13.198472 21397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:05:13.199775 21397 hybrid_clock.cc:648] HybridClock initialized: now 1785985513199691 us; error 72 us; skew 500 ppm
I20260806 03:05:13.201041 21397 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 03:05:13 dist-test-slave-dhph krb5kdc[21383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985513, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:05:13.210772 21397 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 03:05:13.212800 21397 webserver.cc:460] Webserver started at http://127.0.0.1:40257/ using document root <none> and password file <none>
I20260806 03:05:13.213599 21397 fs_manager.cc:359] Metadata directory not provided
I20260806 03:05:13.213766 21397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:05:13.214437 21397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:05:13.218420 21397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785985105255475-32043-0/minicluster-data/master-0/data/instance:
uuid: "c19984e95f504aaf87a247f34fbfd398"
format_stamp: "Formatted at 2026-08-06 03:05:13 on dist-test-slave-dhph"
server_key: "4a1b14926003527b87aad8b84afd923f"
server_key_iv: "c2448a85de1553383aa1bc3c17a067b7"
server_key_version: "encryptionkey@0"
I20260806 03:05:13.220247 21397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785985105255475-32043-0/minicluster-data/master-0/wal/instance:
uuid: "c19984e95f504aaf87a247f34fbfd398"
format_stamp: "Formatted at 2026-08-06 03:05:13 on dist-test-slave-dhph"
server_key: "4a1b14926003527b87aad8b84afd923f"
server_key_iv: "c2448a85de1553383aa1bc3c17a067b7"
server_key_version: "encryptionkey@0"
I20260806 03:05:13.226568 21397 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.004s
I20260806 03:05:13.231240 21413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:05:13.232954 21397 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260806 03:05:13.233182 21397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785985105255475-32043-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785985105255475-32043-0/minicluster-data/master-0/wal
uuid: "c19984e95f504aaf87a247f34fbfd398"
format_stamp: "Formatted at 2026-08-06 03:05:13 on dist-test-slave-dhph"
server_key: "4a1b14926003527b87aad8b84afd923f"
server_key_iv: "c2448a85de1553383aa1bc3c17a067b7"
server_key_version: "encryptionkey@0"
I20260806 03:05:13.233417 21397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785985105255475-32043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785985105255475-32043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785985105255475-32043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:05:13.252194 21397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:05:13.259039 21397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:05:13.259392 21397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:05:13.283658 21397 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44581
I20260806 03:05:13.283705 21467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44581 every 8 connection(s)
I20260806 03:05:13.285169 21397 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
I20260806 03:05:13.289873 21468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:05:13.291709 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 21397
I20260806 03:05:13.291908 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785985105255475-32043-0/minicluster-data/master-0/wal/instance
I20260806 03:05:13.292366 32043 external_mini_cluster.cc:1428] Setting key 60313eb84a297851ad80f29260d7b815
I20260806 03:05:13.306811 21468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c19984e95f504aaf87a247f34fbfd398: Bootstrap starting.
Aug 06 03:05:13 dist-test-slave-dhph krb5kdc[21383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985512, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 03:05:13.311579 21468 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c19984e95f504aaf87a247f34fbfd398: Neither blocks nor log segments found. Creating new log.
I20260806 03:05:13.312928 21468 log.cc:824] T 00000000000000000000000000000000 P c19984e95f504aaf87a247f34fbfd398: Log is configured to *not* fsync() on all Append() calls
I20260806 03:05:13.317173 21468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c19984e95f504aaf87a247f34fbfd398: No bootstrap required, opened a new log
I20260806 03:05:13.324364 21471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:13.294389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.112:35441 (local address 127.0.0.1:44581)
0806 03:05:13.295303 (+ 914us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:13.295317 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:13.295369 (+ 52us) server_negotiation.cc:407] Connection header received
0806 03:05:13.297092 (+ 1723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:13.297115 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:13.297564 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:13.297931 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:13.300126 (+ 2195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:13.302027 (+ 1901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:13.304563 (+ 2536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:13.305440 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:13.305716 (+ 276us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:05:13.312598 (+ 6882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:05:13.312608 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:05:13.312625 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:05:13.312732 (+ 107us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:05:13.318354 (+ 5622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:05:13.320070 (+ 1716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:05:13.320078 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:05:13.320086 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:05:13.320327 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:05:13.320953 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:05:13.320957 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:05:13.320960 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:05:13.321649 (+ 689us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:05:13.322130 (+ 481us) server_negotiation.cc:1035] Waiting for connection context
0806 03:05:13.322703 (+ 573us) server_negotiation.cc:299] Negotiation successful
0806 03:05:13.323097 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":678,"thread_start_us":461,"threads_started":1}
I20260806 03:05:13.327235 21468 raft_consensus.cc:348] T 00000000000000000000000000000000 P c19984e95f504aaf87a247f34fbfd398 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c19984e95f504aaf87a247f34fbfd398" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44581 } }
I20260806 03:05:13.327831 21468 raft_consensus.cc:374] T 00000000000000000000000000000000 P c19984e95f504aaf87a247f34fbfd398 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:05:13.328028 21468 raft_consensus.cc:729] T 00000000000000000000000000000000 P c19984e95f504aaf87a247f34fbfd398 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c19984e95f504aaf87a247f34fbfd398, State: Initialized, Role: FOLLOWER
I20260806 03:05:13.328780 21468 consensus_queue.cc:260] T 00000000000000000000000000000000 P c19984e95f504aaf87a247f34fbfd398 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c19984e95f504aaf87a247f34fbfd398" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44581 } }
I20260806 03:05:13.329077 21468 raft_consensus.cc:388] T 00000000000000000000000000000000 P c19984e95f504aaf87a247f34fbfd398 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:05:13.329243 21468 raft_consensus.cc:482] T 00000000000000000000000000000000 P c19984e95f504aaf87a247f34fbfd398 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:05:13.329444 21468 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c19984e95f504aaf87a247f34fbfd398 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:05:13.329887 32043 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 03:05:13.332172 21468 raft_consensus.cc:504] T 00000000000000000000000000000000 P c19984e95f504aaf87a247f34fbfd398 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c19984e95f504aaf87a247f34fbfd398" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44581 } }
I20260806 03:05:13.332787 21468 leader_election.cc:302] T 00000000000000000000000000000000 P c19984e95f504aaf87a247f34fbfd398 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c19984e95f504aaf87a247f34fbfd398; no voters:
I20260806 03:05:13.333573 21468 leader_election.cc:288] T 00000000000000000000000000000000 P c19984e95f504aaf87a247f34fbfd398 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:05:13.333707 21474 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c19984e95f504aaf87a247f34fbfd398 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:05:13.334854 21474 raft_consensus.cc:686] T 00000000000000000000000000000000 P c19984e95f504aaf87a247f34fbfd398 [term 1 LEADER]: Becoming Leader. State: Replica: c19984e95f504aaf87a247f34fbfd398, State: Running, Role: LEADER
I20260806 03:05:13.335949 21474 consensus_queue.cc:237] T 00000000000000000000000000000000 P c19984e95f504aaf87a247f34fbfd398 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c19984e95f504aaf87a247f34fbfd398" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44581 } }
I20260806 03:05:13.337787 21468 sys_catalog.cc:564] T 00000000000000000000000000000000 P c19984e95f504aaf87a247f34fbfd398 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:05:13.339671 21476 sys_catalog.cc:455] T 00000000000000000000000000000000 P c19984e95f504aaf87a247f34fbfd398 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c19984e95f504aaf87a247f34fbfd398. Latest consensus state: current_term: 1 leader_uuid: "c19984e95f504aaf87a247f34fbfd398" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c19984e95f504aaf87a247f34fbfd398" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44581 } } }
I20260806 03:05:13.340160 21476 sys_catalog.cc:458] T 00000000000000000000000000000000 P c19984e95f504aaf87a247f34fbfd398 [sys.catalog]: This master's current role is: LEADER
I20260806 03:05:13.341053 21477 sys_catalog.cc:455] T 00000000000000000000000000000000 P c19984e95f504aaf87a247f34fbfd398 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c19984e95f504aaf87a247f34fbfd398" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c19984e95f504aaf87a247f34fbfd398" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44581 } } }
I20260806 03:05:13.341502 21477 sys_catalog.cc:458] T 00000000000000000000000000000000 P c19984e95f504aaf87a247f34fbfd398 [sys.catalog]: This master's current role is: LEADER
I20260806 03:05:13.356145 21483 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:05:13.363206 21471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:13.339221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.112:52497 (local address 127.0.0.1:44581)
0806 03:05:13.340164 (+ 943us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:13.340172 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:13.340232 (+ 60us) server_negotiation.cc:407] Connection header received
0806 03:05:13.340504 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:13.340509 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:13.340597 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:13.340707 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:13.343481 (+ 2774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:13.345175 (+ 1694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:13.348395 (+ 3220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:13.348904 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:13.349067 (+ 163us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:05:13.353364 (+ 4297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:05:13.353374 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:05:13.353382 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:05:13.353447 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:05:13.357598 (+ 4151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:05:13.359076 (+ 1478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:05:13.359085 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:05:13.359091 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:05:13.359263 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:05:13.359899 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:05:13.359904 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:05:13.360039 (+ 135us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:05:13.360867 (+ 828us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:05:13.361672 (+ 805us) server_negotiation.cc:1035] Waiting for connection context
0806 03:05:13.362476 (+ 804us) server_negotiation.cc:299] Negotiation successful
0806 03:05:13.362767 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":817}
I20260806 03:05:13.364423 21483 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:05:13.375253 21483 catalog_manager.cc:1347] Generated new cluster ID: 3e593d9fa2f54be19e9fb6b170cedeca
I20260806 03:05:13.375419 21483 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:05:13.400714 21483 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:05:13.403142 21483 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:05:13.421177 21483 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c19984e95f504aaf87a247f34fbfd398: Generated new TSK 0
I20260806 03:05:13.422211 21483 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 03:05:13.436162 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 21397
I20260806 03:05:13.438019 21463 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 03:05:13.545313 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 21397
W20260806 03:05:13.565733 21381 connection.cc:535] client connection to 127.0.0.1:44581 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44581): Connection reset by peer (error 104)
2026-08-06T03:05:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2495 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785985105255475-32043-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 03:05:13 dist-test-slave-dhph krb5kdc[21502](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 03:05:13 dist-test-slave-dhph krb5kdc[21502](info): set up 2 sockets
Aug 06 03:05:13 dist-test-slave-dhph krb5kdc[21502](info): commencing operation
krb5kdc: starting...
W20260806 03:05:14.860152 32043 mini_kdc.cc:121] Time spent starting KDC: real 1.273s user 0.000s sys 0.059s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785985105255475-32043-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785985105255475-32043-0/krb5kdc/test-user.keytab.
Aug 06 03:05:15 dist-test-slave-dhph krb5kdc[21502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T03:05:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:05:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785985105255475-32043-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785985105255475-32043-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 03:05:15.295835 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785985105255475-32043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785985105255475-32043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785985105255475-32043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37649
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785985105255475-32043-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37649
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:05:15.691830 21517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:05:15.692158 21517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:05:15.692256 21517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:05:15.700310 21517 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:05:15.700460 21517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:05:15.700634 21517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:05:15.700749 21517 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:05:15.700841 21517 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:05:15.711745 21517 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785985105255475-32043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785985105255475-32043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37649
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785985105255475-32043-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785985105255475-32043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:05:15.713418 21517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:05:15.714958 21517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:05:15.729735 21522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:05:15.729761 21525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:05:15.729969 21523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:05:15.731038 21517 server_base.cc:1034] running on GCE node
I20260806 03:05:15.732125 21517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:05:15.733846 21517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:05:15.735098 21517 hybrid_clock.cc:648] HybridClock initialized: now 1785985515735030 us; error 69 us; skew 500 ppm
I20260806 03:05:15.735512 21517 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 03:05:15 dist-test-slave-dhph krb5kdc[21502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985515, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:05:15.738987 21517 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 03:05:15.740614 21517 webserver.cc:460] Webserver started at http://127.0.0.1:41257/ using document root <none> and password file <none>
I20260806 03:05:15.741333 21517 fs_manager.cc:359] Metadata directory not provided
I20260806 03:05:15.741463 21517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:05:15.741794 21517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:05:15.745024 21517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785985105255475-32043-0/minicluster-data/master-0/data/instance:
uuid: "38f086d20e4a44e0a8e1e691d0a0d8af"
format_stamp: "Formatted at 2026-08-06 03:05:15 on dist-test-slave-dhph"
server_key: "c960c70f1563ae93eabfc32692eac62a"
server_key_iv: "244a84bbbba4ea3fc8c254549f433c11"
server_key_version: "encryptionkey@0"
I20260806 03:05:15.745904 21517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785985105255475-32043-0/minicluster-data/master-0/wal/instance:
uuid: "38f086d20e4a44e0a8e1e691d0a0d8af"
format_stamp: "Formatted at 2026-08-06 03:05:15 on dist-test-slave-dhph"
server_key: "c960c70f1563ae93eabfc32692eac62a"
server_key_iv: "244a84bbbba4ea3fc8c254549f433c11"
server_key_version: "encryptionkey@0"
I20260806 03:05:15.751884 21517 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20260806 03:05:15.756430 21533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:05:15.758268 21517 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260806 03:05:15.758503 21517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785985105255475-32043-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785985105255475-32043-0/minicluster-data/master-0/wal
uuid: "38f086d20e4a44e0a8e1e691d0a0d8af"
format_stamp: "Formatted at 2026-08-06 03:05:15 on dist-test-slave-dhph"
server_key: "c960c70f1563ae93eabfc32692eac62a"
server_key_iv: "244a84bbbba4ea3fc8c254549f433c11"
server_key_version: "encryptionkey@0"
I20260806 03:05:15.758816 21517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785985105255475-32043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785985105255475-32043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785985105255475-32043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:05:15.778820 21517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:05:15.784400 21517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:05:15.784807 21517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:05:15.811358 21586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37649 every 8 connection(s)
I20260806 03:05:15.811376 21517 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37649
I20260806 03:05:15.813122 21517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
I20260806 03:05:15.816490 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 21517
I20260806 03:05:15.816691 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785985105255475-32043-0/minicluster-data/master-0/wal/instance
I20260806 03:05:15.817245 32043 external_mini_cluster.cc:1428] Setting key e34aed253f4984b9c095e90cb8c0ec00
I20260806 03:05:15.818121 21587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:05:15.845402 21590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:15.819450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.112:43261 (local address 127.0.0.1:37649)
0806 03:05:15.821262 (+ 1812us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:15.821274 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:15.821320 (+ 46us) server_negotiation.cc:407] Connection header received
0806 03:05:15.824545 (+ 3225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:15.824571 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:15.825261 (+ 690us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:15.825646 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:15.826908 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:15.829715 (+ 2807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:15.832432 (+ 2717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:15.834881 (+ 2449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:15.836655 (+ 1774us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:05:15.841753 (+ 5098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:05:15.841764 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:05:15.841783 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:05:15.843060 (+ 1277us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:05:15.843165 (+ 105us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:05:15.843214 (+ 49us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:05:15.843375 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0806 03:05:15.843958 (+ 583us) server_negotiation.cc:299] Negotiation successful
0806 03:05:15.844184 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1583,"thread_start_us":828,"threads_started":1}
I20260806 03:05:15.846091 21587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38f086d20e4a44e0a8e1e691d0a0d8af: Bootstrap starting.
I20260806 03:05:15.850358 32043 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 03:05:15.851032 21587 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 38f086d20e4a44e0a8e1e691d0a0d8af: Neither blocks nor log segments found. Creating new log.
I20260806 03:05:15.852401 21587 log.cc:824] T 00000000000000000000000000000000 P 38f086d20e4a44e0a8e1e691d0a0d8af: Log is configured to *not* fsync() on all Append() calls
I20260806 03:05:15.860908 21587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38f086d20e4a44e0a8e1e691d0a0d8af: No bootstrap required, opened a new log
I20260806 03:05:15.865962 21590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:15.856614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.112:33539 (local address 127.0.0.1:37649)
0806 03:05:15.856847 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:15.856854 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:15.857730 (+ 876us) server_negotiation.cc:407] Connection header received
0806 03:05:15.858059 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:15.858069 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:15.858141 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:15.858257 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:15.858968 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:15.860832 (+ 1864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:15.862517 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:15.863212 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:15.863392 (+ 180us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:05:15.864944 (+ 1552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:05:15.864955 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:05:15.864959 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:05:15.864983 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:05:15.865033 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:05:15.865053 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:05:15.865315 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0806 03:05:15.865533 (+ 218us) server_negotiation.cc:299] Negotiation successful
0806 03:05:15.865602 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260806 03:05:15.871232 21587 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38f086d20e4a44e0a8e1e691d0a0d8af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38f086d20e4a44e0a8e1e691d0a0d8af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37649 } }
I20260806 03:05:15.871706 21587 raft_consensus.cc:374] T 00000000000000000000000000000000 P 38f086d20e4a44e0a8e1e691d0a0d8af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:05:15.871829 21587 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38f086d20e4a44e0a8e1e691d0a0d8af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38f086d20e4a44e0a8e1e691d0a0d8af, State: Initialized, Role: FOLLOWER
I20260806 03:05:15.872534 21587 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38f086d20e4a44e0a8e1e691d0a0d8af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38f086d20e4a44e0a8e1e691d0a0d8af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37649 } }
I20260806 03:05:15.872822 21587 raft_consensus.cc:388] T 00000000000000000000000000000000 P 38f086d20e4a44e0a8e1e691d0a0d8af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:05:15.872973 21587 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38f086d20e4a44e0a8e1e691d0a0d8af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:05:15.873167 21587 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38f086d20e4a44e0a8e1e691d0a0d8af [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:05:15.877158 21587 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38f086d20e4a44e0a8e1e691d0a0d8af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38f086d20e4a44e0a8e1e691d0a0d8af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37649 } }
I20260806 03:05:15.877852 21587 leader_election.cc:302] T 00000000000000000000000000000000 P 38f086d20e4a44e0a8e1e691d0a0d8af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38f086d20e4a44e0a8e1e691d0a0d8af; no voters:
I20260806 03:05:15.878556 21587 leader_election.cc:288] T 00000000000000000000000000000000 P 38f086d20e4a44e0a8e1e691d0a0d8af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:05:15.878865 21597 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38f086d20e4a44e0a8e1e691d0a0d8af [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:05:15.879643 21597 raft_consensus.cc:686] T 00000000000000000000000000000000 P 38f086d20e4a44e0a8e1e691d0a0d8af [term 1 LEADER]: Becoming Leader. State: Replica: 38f086d20e4a44e0a8e1e691d0a0d8af, State: Running, Role: LEADER
I20260806 03:05:15.880355 21597 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38f086d20e4a44e0a8e1e691d0a0d8af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38f086d20e4a44e0a8e1e691d0a0d8af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37649 } }
I20260806 03:05:15.880996 21587 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38f086d20e4a44e0a8e1e691d0a0d8af [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:05:15.884536 21598 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38f086d20e4a44e0a8e1e691d0a0d8af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "38f086d20e4a44e0a8e1e691d0a0d8af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38f086d20e4a44e0a8e1e691d0a0d8af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37649 } } }
I20260806 03:05:15.885016 21598 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38f086d20e4a44e0a8e1e691d0a0d8af [sys.catalog]: This master's current role is: LEADER
I20260806 03:05:15.885043 21599 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38f086d20e4a44e0a8e1e691d0a0d8af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38f086d20e4a44e0a8e1e691d0a0d8af. Latest consensus state: current_term: 1 leader_uuid: "38f086d20e4a44e0a8e1e691d0a0d8af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38f086d20e4a44e0a8e1e691d0a0d8af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37649 } } }
I20260806 03:05:15.885449 21599 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38f086d20e4a44e0a8e1e691d0a0d8af [sys.catalog]: This master's current role is: LEADER
I20260806 03:05:15.888679 21605 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:05:15.894958 21605 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:05:15.911142 21605 catalog_manager.cc:1347] Generated new cluster ID: 90b0d06f51134bd6a67f7436af048aa2
I20260806 03:05:15.911309 21605 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:05:15.930112 21605 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:05:15.940241 21605 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:05:15.949617 21605 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 38f086d20e4a44e0a8e1e691d0a0d8af: Generated new TSK 0
I20260806 03:05:15.950726 21605 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 03:05:15.959887 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 21517
W20260806 03:05:15.961596 21582 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.112:43261
W20260806 03:05:15.962560 32043 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 21517: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 03:05:15.962767 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 21517
W20260806 03:05:15.984803 21499 connection.cc:535] client connection to 127.0.0.1:37649 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37649): Connection reset by peer (error 104)
2026-08-06T03:05:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2418 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785985105255475-32043-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 03:05:16 dist-test-slave-dhph krb5kdc[21619](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 03:05:16 dist-test-slave-dhph krb5kdc[21619](info): set up 2 sockets
Aug 06 03:05:16 dist-test-slave-dhph krb5kdc[21619](info): commencing operation
krb5kdc: starting...
W20260806 03:05:17.259944 32043 mini_kdc.cc:121] Time spent starting KDC: real 1.254s user 0.002s sys 0.068s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785985105255475-32043-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785985105255475-32043-0/krb5kdc/test-user.keytab.
Aug 06 03:05:17 dist-test-slave-dhph krb5kdc[21619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T03:05:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:05:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785985105255475-32043-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785985105255475-32043-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 03:05:17.662182 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785985105255475-32043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785985105255475-32043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785985105255475-32043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43633
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785985105255475-32043-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43633
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:05:18.057518 21632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:05:18.057947 21632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:05:18.058200 21632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:05:18.066962 21632 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:05:18.067112 21632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:05:18.067288 21632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:05:18.067420 21632 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:05:18.067498 21632 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:05:18.079595 21632 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785985105255475-32043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785985105255475-32043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43633
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785985105255475-32043-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785985105255475-32043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:05:18.081048 21632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:05:18.082456 21632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:05:18.096927 21638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:05:18.096973 21639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:05:18.097178 21641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:05:18.097340 21632 server_base.cc:1034] running on GCE node
I20260806 03:05:18.098443 21632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:05:18.100062 21632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:05:18.101300 21632 hybrid_clock.cc:648] HybridClock initialized: now 1785985518101247 us; error 32 us; skew 500 ppm
I20260806 03:05:18.101733 21632 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 03:05:18 dist-test-slave-dhph krb5kdc[21619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985518, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:05:18.105347 21632 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 03:05:18.107158 21632 webserver.cc:460] Webserver started at http://127.0.0.1:37839/ using document root <none> and password file <none>
I20260806 03:05:18.107873 21632 fs_manager.cc:359] Metadata directory not provided
I20260806 03:05:18.108008 21632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:05:18.108343 21632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:05:18.111507 21632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785985105255475-32043-0/minicluster-data/master-0/data/instance:
uuid: "39c992b944c84640a7a4c1684a9c9dad"
format_stamp: "Formatted at 2026-08-06 03:05:18 on dist-test-slave-dhph"
server_key: "a1f74a485cc8f343c2515cd49df79290"
server_key_iv: "4b59d8bbc9eefaadfb7adf5dfa0005f3"
server_key_version: "encryptionkey@0"
I20260806 03:05:18.112551 21632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785985105255475-32043-0/minicluster-data/master-0/wal/instance:
uuid: "39c992b944c84640a7a4c1684a9c9dad"
format_stamp: "Formatted at 2026-08-06 03:05:18 on dist-test-slave-dhph"
server_key: "a1f74a485cc8f343c2515cd49df79290"
server_key_iv: "4b59d8bbc9eefaadfb7adf5dfa0005f3"
server_key_version: "encryptionkey@0"
I20260806 03:05:18.120325 21632 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.004s
I20260806 03:05:18.125052 21648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:05:18.126959 21632 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260806 03:05:18.127220 21632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785985105255475-32043-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785985105255475-32043-0/minicluster-data/master-0/wal
uuid: "39c992b944c84640a7a4c1684a9c9dad"
format_stamp: "Formatted at 2026-08-06 03:05:18 on dist-test-slave-dhph"
server_key: "a1f74a485cc8f343c2515cd49df79290"
server_key_iv: "4b59d8bbc9eefaadfb7adf5dfa0005f3"
server_key_version: "encryptionkey@0"
I20260806 03:05:18.127475 21632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785985105255475-32043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785985105255475-32043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785985105255475-32043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:05:18.134562 21632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:05:18.140444 21632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:05:18.140852 21632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:05:18.166679 21632 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43633
I20260806 03:05:18.166702 21699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43633 every 8 connection(s)
I20260806 03:05:18.168315 21632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
I20260806 03:05:18.172348 21700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:05:18.173362 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 21632
I20260806 03:05:18.173559 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785985105255475-32043-0/minicluster-data/master-0/wal/instance
I20260806 03:05:18.174029 32043 external_mini_cluster.cc:1428] Setting key 8bdd606276e2d969e87b76feb7ddb8ba
I20260806 03:05:18.183624 21703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:18.175994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.112:36459 (local address 127.0.0.1:43633)
0806 03:05:18.176891 (+ 897us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:18.176907 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:18.176953 (+ 46us) server_negotiation.cc:407] Connection header received
0806 03:05:18.178789 (+ 1836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:18.178812 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:18.179385 (+ 573us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:18.179913 (+ 528us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:18.181171 (+ 1258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:05:18.181179 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:05:18.181194 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:05:18.181213 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:05:18.181285 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:05:18.181308 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:05:18.181603 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0806 03:05:18.181767 (+ 164us) server_negotiation.cc:299] Negotiation successful
0806 03:05:18.182000 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":399,"threads_started":1}
I20260806 03:05:18.186890 21700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39c992b944c84640a7a4c1684a9c9dad: Bootstrap starting.
I20260806 03:05:18.188104 32043 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 03:05:18.190215 21700 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 39c992b944c84640a7a4c1684a9c9dad: Neither blocks nor log segments found. Creating new log.
I20260806 03:05:18.191773 21700 log.cc:824] T 00000000000000000000000000000000 P 39c992b944c84640a7a4c1684a9c9dad: Log is configured to *not* fsync() on all Append() calls
I20260806 03:05:18.196841 21700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39c992b944c84640a7a4c1684a9c9dad: No bootstrap required, opened a new log
I20260806 03:05:18.197376 21703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:18.193826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.112:57197 (local address 127.0.0.1:43633)
0806 03:05:18.194196 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:18.194204 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:18.194651 (+ 447us) server_negotiation.cc:407] Connection header received
0806 03:05:18.194852 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:18.194860 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:18.194953 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:18.195168 (+ 215us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:18.196536 (+ 1368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:05:18.196543 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:05:18.196548 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:05:18.196574 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:05:18.196637 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:05:18.196663 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:05:18.196823 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0806 03:05:18.196969 (+ 146us) server_negotiation.cc:299] Negotiation successful
0806 03:05:18.197049 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260806 03:05:18.205770 21700 raft_consensus.cc:348] T 00000000000000000000000000000000 P 39c992b944c84640a7a4c1684a9c9dad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39c992b944c84640a7a4c1684a9c9dad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43633 } }
I20260806 03:05:18.206244 21700 raft_consensus.cc:374] T 00000000000000000000000000000000 P 39c992b944c84640a7a4c1684a9c9dad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:05:18.206399 21700 raft_consensus.cc:729] T 00000000000000000000000000000000 P 39c992b944c84640a7a4c1684a9c9dad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39c992b944c84640a7a4c1684a9c9dad, State: Initialized, Role: FOLLOWER
I20260806 03:05:18.207185 21700 consensus_queue.cc:260] T 00000000000000000000000000000000 P 39c992b944c84640a7a4c1684a9c9dad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39c992b944c84640a7a4c1684a9c9dad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43633 } }
I20260806 03:05:18.207458 21700 raft_consensus.cc:388] T 00000000000000000000000000000000 P 39c992b944c84640a7a4c1684a9c9dad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:05:18.207635 21700 raft_consensus.cc:482] T 00000000000000000000000000000000 P 39c992b944c84640a7a4c1684a9c9dad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:05:18.207876 21700 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 39c992b944c84640a7a4c1684a9c9dad [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:05:18.210981 21700 raft_consensus.cc:504] T 00000000000000000000000000000000 P 39c992b944c84640a7a4c1684a9c9dad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39c992b944c84640a7a4c1684a9c9dad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43633 } }
I20260806 03:05:18.211567 21700 leader_election.cc:302] T 00000000000000000000000000000000 P 39c992b944c84640a7a4c1684a9c9dad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 39c992b944c84640a7a4c1684a9c9dad; no voters:
I20260806 03:05:18.212464 21700 leader_election.cc:288] T 00000000000000000000000000000000 P 39c992b944c84640a7a4c1684a9c9dad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:05:18.212633 21710 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 39c992b944c84640a7a4c1684a9c9dad [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:05:18.213572 21710 raft_consensus.cc:686] T 00000000000000000000000000000000 P 39c992b944c84640a7a4c1684a9c9dad [term 1 LEADER]: Becoming Leader. State: Replica: 39c992b944c84640a7a4c1684a9c9dad, State: Running, Role: LEADER
I20260806 03:05:18.214277 21710 consensus_queue.cc:237] T 00000000000000000000000000000000 P 39c992b944c84640a7a4c1684a9c9dad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39c992b944c84640a7a4c1684a9c9dad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43633 } }
I20260806 03:05:18.214844 21700 sys_catalog.cc:564] T 00000000000000000000000000000000 P 39c992b944c84640a7a4c1684a9c9dad [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:05:18.218094 21711 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39c992b944c84640a7a4c1684a9c9dad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "39c992b944c84640a7a4c1684a9c9dad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39c992b944c84640a7a4c1684a9c9dad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43633 } } }
I20260806 03:05:18.218525 21711 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39c992b944c84640a7a4c1684a9c9dad [sys.catalog]: This master's current role is: LEADER
I20260806 03:05:18.218279 21712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39c992b944c84640a7a4c1684a9c9dad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 39c992b944c84640a7a4c1684a9c9dad. Latest consensus state: current_term: 1 leader_uuid: "39c992b944c84640a7a4c1684a9c9dad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39c992b944c84640a7a4c1684a9c9dad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43633 } } }
I20260806 03:05:18.218794 21712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39c992b944c84640a7a4c1684a9c9dad [sys.catalog]: This master's current role is: LEADER
I20260806 03:05:18.222146 21717 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:05:18.228338 21717 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:05:18.238732 21717 catalog_manager.cc:1347] Generated new cluster ID: 473c074eb1824beda719fbc62ba329ca
I20260806 03:05:18.238855 21717 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:05:18.265509 21717 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:05:18.266214 21717 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:05:18.287863 21717 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 39c992b944c84640a7a4c1684a9c9dad: Generated new TSK 0
I20260806 03:05:18.289225 21717 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 03:05:18.309338 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 21632
W20260806 03:05:18.311035 21695 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.112:36459
W20260806 03:05:18.311864 32043 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 21632: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 03:05:18.312034 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 21632
2026-08-06T03:05:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (2348 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785985105255475-32043-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 03:05:18 dist-test-slave-dhph krb5kdc[21732](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 03:05:18 dist-test-slave-dhph krb5kdc[21732](info): set up 2 sockets
Aug 06 03:05:18 dist-test-slave-dhph krb5kdc[21732](info): commencing operation
krb5kdc: starting...
W20260806 03:05:19.566772 32043 mini_kdc.cc:121] Time spent starting KDC: real 1.213s user 0.002s sys 0.057s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785985105255475-32043-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785985105255475-32043-0/krb5kdc/test-user.keytab.
Aug 06 03:05:19 dist-test-slave-dhph krb5kdc[21732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985519, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T03:05:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:05:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785985105255475-32043-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785985105255475-32043-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 03:05:19.977082 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785985105255475-32043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785985105255475-32043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785985105255475-32043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41625
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785985105255475-32043-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41625
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:05:20.373485 21745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:05:20.373822 21745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:05:20.373921 21745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:05:20.381441 21745 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:05:20.381592 21745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:05:20.381641 21745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:05:20.381697 21745 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:05:20.381736 21745 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:05:20.393420 21745 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785985105255475-32043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785985105255475-32043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41625
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785985105255475-32043-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785985105255475-32043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:05:20.394812 21745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:05:20.396195 21745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:05:20.410161 21751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:05:20.410053 21752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:05:20.410051 21755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:05:20.411720 21745 server_base.cc:1034] running on GCE node
I20260806 03:05:20.412570 21745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:05:20.414122 21745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:05:20.415390 21745 hybrid_clock.cc:648] HybridClock initialized: now 1785985520415329 us; error 54 us; skew 500 ppm
I20260806 03:05:20.415899 21745 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 03:05:20 dist-test-slave-dhph krb5kdc[21732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985520, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:05:20.419731 21745 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 03:05:20.422317 21745 webserver.cc:460] Webserver started at http://127.0.0.1:35241/ using document root <none> and password file <none>
I20260806 03:05:20.423420 21745 fs_manager.cc:359] Metadata directory not provided
I20260806 03:05:20.423645 21745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:05:20.424028 21745 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:05:20.427699 21745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785985105255475-32043-0/minicluster-data/master-0/data/instance:
uuid: "3b5be5b2fe83413caa023d8a5ae6b502"
format_stamp: "Formatted at 2026-08-06 03:05:20 on dist-test-slave-dhph"
server_key: "f99ebadf4f2156096ebaef5c4dcb8939"
server_key_iv: "674ae8203bf2cc3348da37fc2cf5adbf"
server_key_version: "encryptionkey@0"
I20260806 03:05:20.428697 21745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785985105255475-32043-0/minicluster-data/master-0/wal/instance:
uuid: "3b5be5b2fe83413caa023d8a5ae6b502"
format_stamp: "Formatted at 2026-08-06 03:05:20 on dist-test-slave-dhph"
server_key: "f99ebadf4f2156096ebaef5c4dcb8939"
server_key_iv: "674ae8203bf2cc3348da37fc2cf5adbf"
server_key_version: "encryptionkey@0"
I20260806 03:05:20.434962 21745 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260806 03:05:20.438763 21762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:05:20.440505 21745 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 03:05:20.440779 21745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785985105255475-32043-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785985105255475-32043-0/minicluster-data/master-0/wal
uuid: "3b5be5b2fe83413caa023d8a5ae6b502"
format_stamp: "Formatted at 2026-08-06 03:05:20 on dist-test-slave-dhph"
server_key: "f99ebadf4f2156096ebaef5c4dcb8939"
server_key_iv: "674ae8203bf2cc3348da37fc2cf5adbf"
server_key_version: "encryptionkey@0"
I20260806 03:05:20.441013 21745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785985105255475-32043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785985105255475-32043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785985105255475-32043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:05:20.466269 21745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:05:20.471956 21745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:05:20.472338 21745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:05:20.494431 21745 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41625
I20260806 03:05:20.494477 21815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41625 every 8 connection(s)
I20260806 03:05:20.496059 21745 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
I20260806 03:05:20.500516 21816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:05:20.500509 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 21745
I20260806 03:05:20.500715 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785985105255475-32043-0/minicluster-data/master-0/wal/instance
I20260806 03:05:20.501278 32043 external_mini_cluster.cc:1428] Setting key d3b490f5650b7c234490c57667e1a313
I20260806 03:05:20.516568 21816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b5be5b2fe83413caa023d8a5ae6b502: Bootstrap starting.
Aug 06 03:05:20 dist-test-slave-dhph krb5kdc[21732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985519, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 03:05:20.520874 21816 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b5be5b2fe83413caa023d8a5ae6b502: Neither blocks nor log segments found. Creating new log.
I20260806 03:05:20.522372 21816 log.cc:824] T 00000000000000000000000000000000 P 3b5be5b2fe83413caa023d8a5ae6b502: Log is configured to *not* fsync() on all Append() calls
I20260806 03:05:20.526383 21816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b5be5b2fe83413caa023d8a5ae6b502: No bootstrap required, opened a new log
I20260806 03:05:20.531553 21819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:20.503302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.112:32987 (local address 127.0.0.1:41625)
0806 03:05:20.504309 (+ 1007us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:20.504322 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:20.504374 (+ 52us) server_negotiation.cc:407] Connection header received
0806 03:05:20.507103 (+ 2729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:20.507131 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:20.507633 (+ 502us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:20.508066 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:20.510104 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:20.511656 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:20.513711 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:20.514503 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:20.514732 (+ 229us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:05:20.520930 (+ 6198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:05:20.520943 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:05:20.520964 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:05:20.521015 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:05:20.525420 (+ 4405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:05:20.526564 (+ 1144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:05:20.526573 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:05:20.526622 (+ 49us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:05:20.526886 (+ 264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:05:20.527955 (+ 1069us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:05:20.527961 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:05:20.527964 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:05:20.528705 (+ 741us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:05:20.529064 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0806 03:05:20.529680 (+ 616us) server_negotiation.cc:299] Negotiation successful
0806 03:05:20.530061 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":747,"thread_start_us":493,"threads_started":1}
I20260806 03:05:20.536635 32043 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 03:05:20.536659 21816 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b5be5b2fe83413caa023d8a5ae6b502 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b5be5b2fe83413caa023d8a5ae6b502" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41625 } }
I20260806 03:05:20.537089 21816 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b5be5b2fe83413caa023d8a5ae6b502 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:05:20.537220 21816 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b5be5b2fe83413caa023d8a5ae6b502 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5be5b2fe83413caa023d8a5ae6b502, State: Initialized, Role: FOLLOWER
I20260806 03:05:20.537837 21816 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b5be5b2fe83413caa023d8a5ae6b502 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b5be5b2fe83413caa023d8a5ae6b502" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41625 } }
I20260806 03:05:20.538075 21816 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b5be5b2fe83413caa023d8a5ae6b502 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:05:20.538235 21816 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b5be5b2fe83413caa023d8a5ae6b502 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:05:20.538405 21816 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b5be5b2fe83413caa023d8a5ae6b502 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:05:20.541028 21816 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b5be5b2fe83413caa023d8a5ae6b502 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b5be5b2fe83413caa023d8a5ae6b502" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41625 } }
I20260806 03:05:20.541540 21816 leader_election.cc:302] T 00000000000000000000000000000000 P 3b5be5b2fe83413caa023d8a5ae6b502 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b5be5b2fe83413caa023d8a5ae6b502; no voters:
I20260806 03:05:20.542486 21816 leader_election.cc:288] T 00000000000000000000000000000000 P 3b5be5b2fe83413caa023d8a5ae6b502 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:05:20.542600 21825 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b5be5b2fe83413caa023d8a5ae6b502 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:05:20.545408 21816 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b5be5b2fe83413caa023d8a5ae6b502 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:05:20.546000 21825 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b5be5b2fe83413caa023d8a5ae6b502 [term 1 LEADER]: Becoming Leader. State: Replica: 3b5be5b2fe83413caa023d8a5ae6b502, State: Running, Role: LEADER
I20260806 03:05:20.546797 21825 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b5be5b2fe83413caa023d8a5ae6b502 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b5be5b2fe83413caa023d8a5ae6b502" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41625 } }
I20260806 03:05:20.551023 21830 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b5be5b2fe83413caa023d8a5ae6b502 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b5be5b2fe83413caa023d8a5ae6b502" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b5be5b2fe83413caa023d8a5ae6b502" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41625 } } }
I20260806 03:05:20.551527 21830 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b5be5b2fe83413caa023d8a5ae6b502 [sys.catalog]: This master's current role is: LEADER
I20260806 03:05:20.552286 21827 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b5be5b2fe83413caa023d8a5ae6b502 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b5be5b2fe83413caa023d8a5ae6b502. Latest consensus state: current_term: 1 leader_uuid: "3b5be5b2fe83413caa023d8a5ae6b502" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b5be5b2fe83413caa023d8a5ae6b502" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41625 } } }
I20260806 03:05:20.552690 21827 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b5be5b2fe83413caa023d8a5ae6b502 [sys.catalog]: This master's current role is: LEADER
I20260806 03:05:20.559628 21834 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:05:20.565497 21834 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:05:20.568225 21819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:20.541943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.112:43123 (local address 127.0.0.1:41625)
0806 03:05:20.542178 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:20.542187 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:20.542820 (+ 633us) server_negotiation.cc:407] Connection header received
0806 03:05:20.543128 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:20.543139 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:20.543246 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:20.543337 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:20.545731 (+ 2394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:20.547336 (+ 1605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:20.549484 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:20.550944 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:20.551144 (+ 200us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:05:20.556350 (+ 5206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:05:20.556365 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:05:20.556375 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:05:20.556443 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:05:20.562083 (+ 5640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:05:20.563691 (+ 1608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:05:20.563700 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:05:20.563706 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:05:20.563941 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:05:20.564755 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:05:20.564763 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:05:20.564771 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:05:20.565385 (+ 614us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:05:20.565734 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0806 03:05:20.567383 (+ 1649us) server_negotiation.cc:299] Negotiation successful
0806 03:05:20.567660 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 03:05:20.578356 21834 catalog_manager.cc:1347] Generated new cluster ID: 79d07b96be2e480685e6c078138d9b11
I20260806 03:05:20.578513 21834 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:05:20.608403 21834 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:05:20.610433 21834 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:05:20.625659 21834 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b5be5b2fe83413caa023d8a5ae6b502: Generated new TSK 0
I20260806 03:05:20.626736 21834 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 03:05:20.638119 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 21745
I20260806 03:05:20.640110 21811 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 03:05:20.739426 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 21745
W20260806 03:05:20.761122 21730 connection.cc:535] client connection to 127.0.0.1:41625 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41625): Connection reset by peer (error 104)
2026-08-06T03:05:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2428 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785985105255475-32043-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 03:05:20 dist-test-slave-dhph krb5kdc[21850](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 03:05:20 dist-test-slave-dhph krb5kdc[21850](info): set up 2 sockets
Aug 06 03:05:20 dist-test-slave-dhph krb5kdc[21850](info): commencing operation
krb5kdc: starting...
W20260806 03:05:22.059134 32043 mini_kdc.cc:121] Time spent starting KDC: real 1.276s user 0.007s sys 0.052s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785985105255475-32043-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785985105255475-32043-0/krb5kdc/test-user.keytab.
Aug 06 03:05:22 dist-test-slave-dhph krb5kdc[21850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T03:05:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:05:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785985105255475-32043-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785985105255475-32043-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 03:05:22.452268 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785985105255475-32043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785985105255475-32043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785985105255475-32043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34079
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785985105255475-32043-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34079
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:05:22.844326 21864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:05:22.844782 21864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:05:22.845112 21864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:05:22.854882 21864 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:05:22.855037 21864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:05:22.855132 21864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:05:22.855252 21864 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:05:22.855319 21864 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:05:22.867026 21864 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785985105255475-32043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785985105255475-32043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34079
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785985105255475-32043-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785985105255475-32043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:05:22.868530 21864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:05:22.870040 21864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:05:22.883392 21872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:05:22.883428 21869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:05:22.883384 21870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:05:22.884315 21864 server_base.cc:1034] running on GCE node
I20260806 03:05:22.885111 21864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:05:22.886723 21864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:05:22.887969 21864 hybrid_clock.cc:648] HybridClock initialized: now 1785985522887904 us; error 55 us; skew 500 ppm
I20260806 03:05:22.888372 21864 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 03:05:22 dist-test-slave-dhph krb5kdc[21850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985522, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:05:22.893646 21864 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 03:05:22.895612 21864 webserver.cc:460] Webserver started at http://127.0.0.1:33853/ using document root <none> and password file <none>
I20260806 03:05:22.896420 21864 fs_manager.cc:359] Metadata directory not provided
I20260806 03:05:22.896584 21864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:05:22.896917 21864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:05:22.900262 21864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785985105255475-32043-0/minicluster-data/master-0/data/instance:
uuid: "6c96e82e6342445f9bb76c09b63c95e0"
format_stamp: "Formatted at 2026-08-06 03:05:22 on dist-test-slave-dhph"
server_key: "fba34ca44015bd4ce90114e8f288351b"
server_key_iv: "db60c2e3008226be2928d03cdbd00b2b"
server_key_version: "encryptionkey@0"
I20260806 03:05:22.901082 21864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785985105255475-32043-0/minicluster-data/master-0/wal/instance:
uuid: "6c96e82e6342445f9bb76c09b63c95e0"
format_stamp: "Formatted at 2026-08-06 03:05:22 on dist-test-slave-dhph"
server_key: "fba34ca44015bd4ce90114e8f288351b"
server_key_iv: "db60c2e3008226be2928d03cdbd00b2b"
server_key_version: "encryptionkey@0"
I20260806 03:05:22.907047 21864 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260806 03:05:22.911628 21880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:05:22.913371 21864 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260806 03:05:22.913609 21864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785985105255475-32043-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785985105255475-32043-0/minicluster-data/master-0/wal
uuid: "6c96e82e6342445f9bb76c09b63c95e0"
format_stamp: "Formatted at 2026-08-06 03:05:22 on dist-test-slave-dhph"
server_key: "fba34ca44015bd4ce90114e8f288351b"
server_key_iv: "db60c2e3008226be2928d03cdbd00b2b"
server_key_version: "encryptionkey@0"
I20260806 03:05:22.913854 21864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785985105255475-32043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785985105255475-32043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785985105255475-32043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:05:22.951347 21864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:05:22.957921 21864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:05:22.958560 21864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:05:22.985625 21864 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34079
I20260806 03:05:22.985723 21934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34079 every 8 connection(s)
I20260806 03:05:22.987360 21864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
I20260806 03:05:22.992017 21935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:05:22.993775 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 21864
I20260806 03:05:22.993971 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785985105255475-32043-0/minicluster-data/master-0/wal/instance
I20260806 03:05:22.994436 32043 external_mini_cluster.cc:1428] Setting key d189668e6a3f9766c32b3ec2d8a21f31
I20260806 03:05:23.009076 21935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c96e82e6342445f9bb76c09b63c95e0: Bootstrap starting.
I20260806 03:05:23.012861 21938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:22.996626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.112:34257 (local address 127.0.0.1:34079)
0806 03:05:22.997543 (+ 917us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:22.997555 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:22.997604 (+ 49us) server_negotiation.cc:407] Connection header received
0806 03:05:22.999350 (+ 1746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:22.999375 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:22.999824 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:23.000181 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:23.000888 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:23.003119 (+ 2231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:23.005417 (+ 2298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:23.006450 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:23.006775 (+ 325us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:05:23.009517 (+ 2742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:05:23.009529 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:05:23.009547 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:05:23.010425 (+ 878us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:05:23.010696 (+ 271us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:05:23.010740 (+ 44us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:05:23.010925 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0806 03:05:23.011391 (+ 466us) server_negotiation.cc:299] Negotiation successful
0806 03:05:23.011633 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":695,"thread_start_us":422,"threads_started":1}
I20260806 03:05:23.014324 21935 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c96e82e6342445f9bb76c09b63c95e0: Neither blocks nor log segments found. Creating new log.
I20260806 03:05:23.015849 21935 log.cc:824] T 00000000000000000000000000000000 P 6c96e82e6342445f9bb76c09b63c95e0: Log is configured to *not* fsync() on all Append() calls
I20260806 03:05:23.017656 32043 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 03:05:23.019623 21935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c96e82e6342445f9bb76c09b63c95e0: No bootstrap required, opened a new log
I20260806 03:05:23.029537 21935 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c96e82e6342445f9bb76c09b63c95e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c96e82e6342445f9bb76c09b63c95e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34079 } }
I20260806 03:05:23.030119 21935 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c96e82e6342445f9bb76c09b63c95e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:05:23.030254 21935 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c96e82e6342445f9bb76c09b63c95e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c96e82e6342445f9bb76c09b63c95e0, State: Initialized, Role: FOLLOWER
I20260806 03:05:23.031255 21935 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c96e82e6342445f9bb76c09b63c95e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c96e82e6342445f9bb76c09b63c95e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34079 } }
I20260806 03:05:23.031618 21938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:23.022892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.112:35989 (local address 127.0.0.1:34079)
0806 03:05:23.023156 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:23.023163 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:23.023559 (+ 396us) server_negotiation.cc:407] Connection header received
0806 03:05:23.023774 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:23.023778 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:23.023850 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:23.023933 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:23.024725 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:23.025658 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:23.027826 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:23.028559 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:23.028738 (+ 179us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:05:23.030321 (+ 1583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:05:23.030330 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:05:23.030337 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:05:23.030366 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:05:23.030430 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:05:23.030624 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:05:23.030813 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0806 03:05:23.031122 (+ 309us) server_negotiation.cc:299] Negotiation successful
0806 03:05:23.031240 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260806 03:05:23.031622 21935 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c96e82e6342445f9bb76c09b63c95e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:05:23.031874 21935 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c96e82e6342445f9bb76c09b63c95e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:05:23.032112 21935 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c96e82e6342445f9bb76c09b63c95e0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:05:23.035110 21935 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c96e82e6342445f9bb76c09b63c95e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c96e82e6342445f9bb76c09b63c95e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34079 } }
I20260806 03:05:23.035689 21935 leader_election.cc:302] T 00000000000000000000000000000000 P 6c96e82e6342445f9bb76c09b63c95e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c96e82e6342445f9bb76c09b63c95e0; no voters:
I20260806 03:05:23.036365 21935 leader_election.cc:288] T 00000000000000000000000000000000 P 6c96e82e6342445f9bb76c09b63c95e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:05:23.036625 21945 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c96e82e6342445f9bb76c09b63c95e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:05:23.038244 21945 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c96e82e6342445f9bb76c09b63c95e0 [term 1 LEADER]: Becoming Leader. State: Replica: 6c96e82e6342445f9bb76c09b63c95e0, State: Running, Role: LEADER
I20260806 03:05:23.038885 21945 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c96e82e6342445f9bb76c09b63c95e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c96e82e6342445f9bb76c09b63c95e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34079 } }
I20260806 03:05:23.039028 21935 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c96e82e6342445f9bb76c09b63c95e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:05:23.042192 21947 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c96e82e6342445f9bb76c09b63c95e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c96e82e6342445f9bb76c09b63c95e0. Latest consensus state: current_term: 1 leader_uuid: "6c96e82e6342445f9bb76c09b63c95e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c96e82e6342445f9bb76c09b63c95e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34079 } } }
I20260806 03:05:23.042645 21947 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c96e82e6342445f9bb76c09b63c95e0 [sys.catalog]: This master's current role is: LEADER
I20260806 03:05:23.043138 21946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c96e82e6342445f9bb76c09b63c95e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c96e82e6342445f9bb76c09b63c95e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c96e82e6342445f9bb76c09b63c95e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34079 } } }
I20260806 03:05:23.043449 21946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c96e82e6342445f9bb76c09b63c95e0 [sys.catalog]: This master's current role is: LEADER
I20260806 03:05:23.047274 21952 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:05:23.052708 21952 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:05:23.062031 21952 catalog_manager.cc:1347] Generated new cluster ID: 7c9c65eee62a4a1391f560b54debc4ab
I20260806 03:05:23.062291 21952 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:05:23.078998 21952 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:05:23.080946 21952 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:05:23.099462 21952 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c96e82e6342445f9bb76c09b63c95e0: Generated new TSK 0
I20260806 03:05:23.100481 21952 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 03:05:23.107892 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 21864
W20260806 03:05:23.109350 21930 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.112:34257
W20260806 03:05:23.110275 32043 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 21864: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 03:05:23.110440 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 21864
W20260806 03:05:23.131219 21848 connection.cc:535] client connection to 127.0.0.1:34079 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34079): Connection reset by peer (error 104)
2026-08-06T03:05:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2368 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785985105255475-32043-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 03:05:23 dist-test-slave-dhph krb5kdc[21966](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 03:05:23 dist-test-slave-dhph krb5kdc[21966](info): set up 2 sockets
Aug 06 03:05:23 dist-test-slave-dhph krb5kdc[21966](info): commencing operation
krb5kdc: starting...
W20260806 03:05:24.365754 32043 mini_kdc.cc:121] Time spent starting KDC: real 1.214s user 0.001s sys 0.061s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785985105255475-32043-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785985105255475-32043-0/krb5kdc/test-user.keytab.
Aug 06 03:05:24 dist-test-slave-dhph krb5kdc[21966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985524, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T03:05:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:05:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785985105255475-32043-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785985105255475-32043-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 03:05:24.759511 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785985105255475-32043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785985105255475-32043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785985105255475-32043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42125
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785985105255475-32043-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42125
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:05:25.166389 21979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:05:25.166777 21979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:05:25.166882 21979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:05:25.176283 21979 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:05:25.176452 21979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:05:25.176571 21979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:05:25.176803 21979 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:05:25.176925 21979 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:05:25.187968 21979 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785985105255475-32043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785985105255475-32043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42125
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785985105255475-32043-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42125
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785985105255475-32043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:05:25.189417 21979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:05:25.191170 21979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:05:25.204195 21986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:05:25.204233 21985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:05:25.204520 21988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:05:25.205607 21979 server_base.cc:1034] running on GCE node
I20260806 03:05:25.206467 21979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:05:25.208318 21979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:05:25.209622 21979 hybrid_clock.cc:648] HybridClock initialized: now 1785985525209570 us; error 49 us; skew 500 ppm
I20260806 03:05:25.210023 21979 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 03:05:25 dist-test-slave-dhph krb5kdc[21966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985525, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:05:25.213963 21979 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 03:05:25.215870 21979 webserver.cc:460] Webserver started at http://127.0.0.1:34399/ using document root <none> and password file <none>
I20260806 03:05:25.216655 21979 fs_manager.cc:359] Metadata directory not provided
I20260806 03:05:25.216773 21979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:05:25.217092 21979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:05:25.220413 21979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785985105255475-32043-0/minicluster-data/master-0/data/instance:
uuid: "fed3af4d3c8a4f91b992beda30c0dbf7"
format_stamp: "Formatted at 2026-08-06 03:05:25 on dist-test-slave-dhph"
server_key: "5c90f5f145dbbb13f0d3f3e8ffa71b0a"
server_key_iv: "18078232ef776b19be81f6d3e11d354f"
server_key_version: "encryptionkey@0"
I20260806 03:05:25.221247 21979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785985105255475-32043-0/minicluster-data/master-0/wal/instance:
uuid: "fed3af4d3c8a4f91b992beda30c0dbf7"
format_stamp: "Formatted at 2026-08-06 03:05:25 on dist-test-slave-dhph"
server_key: "5c90f5f145dbbb13f0d3f3e8ffa71b0a"
server_key_iv: "18078232ef776b19be81f6d3e11d354f"
server_key_version: "encryptionkey@0"
I20260806 03:05:25.229025 21979 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.005s
I20260806 03:05:25.235627 21996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:05:25.237860 21979 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260806 03:05:25.238127 21979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785985105255475-32043-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785985105255475-32043-0/minicluster-data/master-0/wal
uuid: "fed3af4d3c8a4f91b992beda30c0dbf7"
format_stamp: "Formatted at 2026-08-06 03:05:25 on dist-test-slave-dhph"
server_key: "5c90f5f145dbbb13f0d3f3e8ffa71b0a"
server_key_iv: "18078232ef776b19be81f6d3e11d354f"
server_key_version: "encryptionkey@0"
I20260806 03:05:25.238411 21979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785985105255475-32043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785985105255475-32043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785985105255475-32043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:05:25.247062 21979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:05:25.253459 21979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:05:25.254016 21979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:05:25.277516 21979 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42125
I20260806 03:05:25.277568 22050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42125 every 8 connection(s)
I20260806 03:05:25.279181 21979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
I20260806 03:05:25.282357 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 21979
I20260806 03:05:25.282673 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785985105255475-32043-0/minicluster-data/master-0/wal/instance
I20260806 03:05:25.283195 32043 external_mini_cluster.cc:1428] Setting key 76badfdb6ff19139daf9d9c2d58d3120
I20260806 03:05:25.283540 22051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:05:25.292693 22054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:25.285322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.112:47033 (local address 127.0.0.1:42125)
0806 03:05:25.286232 (+ 910us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:25.286243 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:25.286290 (+ 47us) server_negotiation.cc:407] Connection header received
0806 03:05:25.288568 (+ 2278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:25.288593 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:25.289062 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:25.289475 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:25.290884 (+ 1409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:05:25.290891 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:05:25.290908 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:05:25.290929 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:05:25.291001 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:05:25.291035 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:05:25.291243 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0806 03:05:25.291370 (+ 127us) server_negotiation.cc:299] Negotiation successful
0806 03:05:25.291628 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":667,"thread_start_us":415,"threads_started":1}
I20260806 03:05:25.297043 32043 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 03:05:25.299046 22051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fed3af4d3c8a4f91b992beda30c0dbf7: Bootstrap starting.
I20260806 03:05:25.303551 22051 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fed3af4d3c8a4f91b992beda30c0dbf7: Neither blocks nor log segments found. Creating new log.
I20260806 03:05:25.305199 22051 log.cc:824] T 00000000000000000000000000000000 P fed3af4d3c8a4f91b992beda30c0dbf7: Log is configured to *not* fsync() on all Append() calls
I20260806 03:05:25.305542 22054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:25.302671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.112:34995 (local address 127.0.0.1:42125)
0806 03:05:25.302886 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:25.302894 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:25.303281 (+ 387us) server_negotiation.cc:407] Connection header received
0806 03:05:25.303468 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:25.303472 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:25.303529 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:25.303619 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:25.304736 (+ 1117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:05:25.304739 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:05:25.304742 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:05:25.304758 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:05:25.304840 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:05:25.304858 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:05:25.304956 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0806 03:05:25.305114 (+ 158us) server_negotiation.cc:299] Negotiation successful
0806 03:05:25.305165 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260806 03:05:25.308641 22051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fed3af4d3c8a4f91b992beda30c0dbf7: No bootstrap required, opened a new log
I20260806 03:05:25.317652 22051 raft_consensus.cc:348] T 00000000000000000000000000000000 P fed3af4d3c8a4f91b992beda30c0dbf7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fed3af4d3c8a4f91b992beda30c0dbf7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42125 } }
I20260806 03:05:25.318100 22051 raft_consensus.cc:374] T 00000000000000000000000000000000 P fed3af4d3c8a4f91b992beda30c0dbf7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:05:25.318177 22051 raft_consensus.cc:729] T 00000000000000000000000000000000 P fed3af4d3c8a4f91b992beda30c0dbf7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fed3af4d3c8a4f91b992beda30c0dbf7, State: Initialized, Role: FOLLOWER
I20260806 03:05:25.318853 22051 consensus_queue.cc:260] T 00000000000000000000000000000000 P fed3af4d3c8a4f91b992beda30c0dbf7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fed3af4d3c8a4f91b992beda30c0dbf7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42125 } }
I20260806 03:05:25.319100 22051 raft_consensus.cc:388] T 00000000000000000000000000000000 P fed3af4d3c8a4f91b992beda30c0dbf7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:05:25.319217 22051 raft_consensus.cc:482] T 00000000000000000000000000000000 P fed3af4d3c8a4f91b992beda30c0dbf7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:05:25.319540 22051 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fed3af4d3c8a4f91b992beda30c0dbf7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:05:25.322294 22051 raft_consensus.cc:504] T 00000000000000000000000000000000 P fed3af4d3c8a4f91b992beda30c0dbf7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fed3af4d3c8a4f91b992beda30c0dbf7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42125 } }
I20260806 03:05:25.322904 22051 leader_election.cc:302] T 00000000000000000000000000000000 P fed3af4d3c8a4f91b992beda30c0dbf7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fed3af4d3c8a4f91b992beda30c0dbf7; no voters:
I20260806 03:05:25.323683 22051 leader_election.cc:288] T 00000000000000000000000000000000 P fed3af4d3c8a4f91b992beda30c0dbf7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:05:25.323894 22061 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fed3af4d3c8a4f91b992beda30c0dbf7 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:05:25.325248 22061 raft_consensus.cc:686] T 00000000000000000000000000000000 P fed3af4d3c8a4f91b992beda30c0dbf7 [term 1 LEADER]: Becoming Leader. State: Replica: fed3af4d3c8a4f91b992beda30c0dbf7, State: Running, Role: LEADER
I20260806 03:05:25.326032 22061 consensus_queue.cc:237] T 00000000000000000000000000000000 P fed3af4d3c8a4f91b992beda30c0dbf7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fed3af4d3c8a4f91b992beda30c0dbf7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42125 } }
I20260806 03:05:25.326658 22051 sys_catalog.cc:564] T 00000000000000000000000000000000 P fed3af4d3c8a4f91b992beda30c0dbf7 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:05:25.329475 22062 sys_catalog.cc:455] T 00000000000000000000000000000000 P fed3af4d3c8a4f91b992beda30c0dbf7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fed3af4d3c8a4f91b992beda30c0dbf7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fed3af4d3c8a4f91b992beda30c0dbf7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42125 } } }
I20260806 03:05:25.329748 22063 sys_catalog.cc:455] T 00000000000000000000000000000000 P fed3af4d3c8a4f91b992beda30c0dbf7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fed3af4d3c8a4f91b992beda30c0dbf7. Latest consensus state: current_term: 1 leader_uuid: "fed3af4d3c8a4f91b992beda30c0dbf7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fed3af4d3c8a4f91b992beda30c0dbf7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42125 } } }
I20260806 03:05:25.330060 22062 sys_catalog.cc:458] T 00000000000000000000000000000000 P fed3af4d3c8a4f91b992beda30c0dbf7 [sys.catalog]: This master's current role is: LEADER
I20260806 03:05:25.330071 22063 sys_catalog.cc:458] T 00000000000000000000000000000000 P fed3af4d3c8a4f91b992beda30c0dbf7 [sys.catalog]: This master's current role is: LEADER
I20260806 03:05:25.336241 22069 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:05:25.342078 22069 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:05:25.353361 22069 catalog_manager.cc:1347] Generated new cluster ID: 0dd7f097028b4081ba6a83bd3a1b5474
I20260806 03:05:25.353533 22069 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:05:25.376185 22069 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:05:25.376670 22069 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:05:25.384835 22069 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fed3af4d3c8a4f91b992beda30c0dbf7: Generated new TSK 0
I20260806 03:05:25.386070 22069 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 03:05:25.401366 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 21979
W20260806 03:05:25.402773 22045 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.112:47033
W20260806 03:05:25.403637 32043 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 21979: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 03:05:25.403803 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 21979
2026-08-06T03:05:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (2292 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785985105255475-32043-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 03:05:25 dist-test-slave-dhph krb5kdc[22082](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 03:05:25 dist-test-slave-dhph krb5kdc[22082](info): set up 2 sockets
Aug 06 03:05:25 dist-test-slave-dhph krb5kdc[22082](info): commencing operation
krb5kdc: starting...
W20260806 03:05:26.707033 32043 mini_kdc.cc:121] Time spent starting KDC: real 1.263s user 0.004s sys 0.054s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785985105255475-32043-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785985105255475-32043-0/krb5kdc/test-user.keytab.
Aug 06 03:05:26 dist-test-slave-dhph krb5kdc[22082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T03:05:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:05:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.74.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab.
Entry for principal kudu/127.31.74.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab.
I20260806 03:05:27.124693 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785985105255475-32043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785985105255475-32043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785985105255475-32043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:33827
--webserver_interface=127.31.74.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab
--principal=kudu/127.31.74.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:40879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.74.254:33827
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:05:27.516690 22095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:05:27.517118 22095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:05:27.517313 22095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:05:27.525525 22095 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:05:27.525715 22095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:05:27.525846 22095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:05:27.525954 22095 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:05:27.526046 22095 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:05:27.537866 22095 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:40879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785985105255475-32043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785985105255475-32043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.74.254:33827
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab
--principal=kudu/127.31.74.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:33827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.74.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785985105255475-32043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:05:27.539427 22095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:05:27.541060 22095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:05:27.555473 22103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:05:27.555464 22101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:05:27.555507 22100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:05:27.556355 22095 server_base.cc:1034] running on GCE node
I20260806 03:05:27.557232 22095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:05:27.558883 22095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:05:27.560148 22095 hybrid_clock.cc:648] HybridClock initialized: now 1785985527560062 us; error 76 us; skew 500 ppm
I20260806 03:05:27.560653 22095 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 03:05:27 dist-test-slave-dhph krb5kdc[22082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985527, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:05:27.564021 22095 init.cc:377] Logged in from keytab as kudu/127.31.74.254@KRBTEST.COM (short username kudu)
I20260806 03:05:27.565666 22095 webserver.cc:460] Webserver started at http://127.31.74.254:38103/ using document root <none> and password file <none>
I20260806 03:05:27.566431 22095 fs_manager.cc:359] Metadata directory not provided
I20260806 03:05:27.566568 22095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:05:27.566922 22095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:05:27.570009 22095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785985105255475-32043-0/minicluster-data/master-0/data/instance:
uuid: "6e0c5b22896f4a6fa14fa9510f294d87"
format_stamp: "Formatted at 2026-08-06 03:05:27 on dist-test-slave-dhph"
server_key: "4032c89117a5fc7371e2310009dcb25c"
server_key_iv: "33c19a354e12bb488a97c0886217c48a"
server_key_version: "encryptionkey@0"
I20260806 03:05:27.571072 22095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785985105255475-32043-0/minicluster-data/master-0/wal/instance:
uuid: "6e0c5b22896f4a6fa14fa9510f294d87"
format_stamp: "Formatted at 2026-08-06 03:05:27 on dist-test-slave-dhph"
server_key: "4032c89117a5fc7371e2310009dcb25c"
server_key_iv: "33c19a354e12bb488a97c0886217c48a"
server_key_version: "encryptionkey@0"
I20260806 03:05:27.577653 22095 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260806 03:05:27.581614 22110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:05:27.583328 22095 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 03:05:27.583561 22095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785985105255475-32043-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785985105255475-32043-0/minicluster-data/master-0/wal
uuid: "6e0c5b22896f4a6fa14fa9510f294d87"
format_stamp: "Formatted at 2026-08-06 03:05:27 on dist-test-slave-dhph"
server_key: "4032c89117a5fc7371e2310009dcb25c"
server_key_iv: "33c19a354e12bb488a97c0886217c48a"
server_key_version: "encryptionkey@0"
I20260806 03:05:27.583814 22095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785985105255475-32043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785985105255475-32043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785985105255475-32043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:05:27.600009 22095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:05:27.604776 22095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:05:27.605305 22095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:05:27.627986 22095 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.254:33827
I20260806 03:05:27.628017 22162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.254:33827 every 8 connection(s)
I20260806 03:05:27.629441 22095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
I20260806 03:05:27.633713 22163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:05:27.637662 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 22095
I20260806 03:05:27.638037 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785985105255475-32043-0/minicluster-data/master-0/wal/instance
I20260806 03:05:27.638505 32043 external_mini_cluster.cc:1428] Setting key 6a18e2bb3d8fd6595bc81b2a23f69876
I20260806 03:05:27.648451 22163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e0c5b22896f4a6fa14fa9510f294d87: Bootstrap starting.
I20260806 03:05:27.652185 22163 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e0c5b22896f4a6fa14fa9510f294d87: Neither blocks nor log segments found. Creating new log.
I20260806 03:05:27.653769 22163 log.cc:824] T 00000000000000000000000000000000 P 6e0c5b22896f4a6fa14fa9510f294d87: Log is configured to *not* fsync() on all Append() calls
Aug 06 03:05:27 dist-test-slave-dhph krb5kdc[22082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 03:05:27.657120 22163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e0c5b22896f4a6fa14fa9510f294d87: No bootstrap required, opened a new log
I20260806 03:05:27.665732 22166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:27.640887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48795 (local address 127.31.74.254:33827)
0806 03:05:27.641753 (+ 866us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:27.641764 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:27.641808 (+ 44us) server_negotiation.cc:407] Connection header received
0806 03:05:27.643641 (+ 1833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:27.643665 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:27.644107 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:27.644493 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:27.646758 (+ 2265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:27.648522 (+ 1764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:27.650507 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:27.651607 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:27.651917 (+ 310us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:05:27.656566 (+ 4649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:05:27.656575 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:05:27.656590 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:05:27.656629 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:05:27.661049 (+ 4420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:05:27.662018 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:05:27.662027 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:05:27.662033 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:05:27.662216 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:05:27.662760 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:05:27.662764 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:05:27.662769 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:05:27.663378 (+ 609us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:05:27.663746 (+ 368us) server_negotiation.cc:1035] Waiting for connection context
0806 03:05:27.664276 (+ 530us) server_negotiation.cc:299] Negotiation successful
0806 03:05:27.664629 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":411,"threads_started":1}
I20260806 03:05:27.666988 22163 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e0c5b22896f4a6fa14fa9510f294d87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e0c5b22896f4a6fa14fa9510f294d87" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 33827 } }
I20260806 03:05:27.667369 22163 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e0c5b22896f4a6fa14fa9510f294d87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:05:27.667497 22163 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e0c5b22896f4a6fa14fa9510f294d87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e0c5b22896f4a6fa14fa9510f294d87, State: Initialized, Role: FOLLOWER
I20260806 03:05:27.668097 22163 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e0c5b22896f4a6fa14fa9510f294d87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e0c5b22896f4a6fa14fa9510f294d87" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 33827 } }
I20260806 03:05:27.668375 22163 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e0c5b22896f4a6fa14fa9510f294d87 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:05:27.668588 22163 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e0c5b22896f4a6fa14fa9510f294d87 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:05:27.668807 22163 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e0c5b22896f4a6fa14fa9510f294d87 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:05:27.670291 32043 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 03:05:27.671497 22163 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e0c5b22896f4a6fa14fa9510f294d87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e0c5b22896f4a6fa14fa9510f294d87" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 33827 } }
I20260806 03:05:27.672021 22163 leader_election.cc:302] T 00000000000000000000000000000000 P 6e0c5b22896f4a6fa14fa9510f294d87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e0c5b22896f4a6fa14fa9510f294d87; no voters:
I20260806 03:05:27.672691 22163 leader_election.cc:288] T 00000000000000000000000000000000 P 6e0c5b22896f4a6fa14fa9510f294d87 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:05:27.672821 22168 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e0c5b22896f4a6fa14fa9510f294d87 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:05:27.674986 22168 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e0c5b22896f4a6fa14fa9510f294d87 [term 1 LEADER]: Becoming Leader. State: Replica: 6e0c5b22896f4a6fa14fa9510f294d87, State: Running, Role: LEADER
I20260806 03:05:27.675772 22163 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e0c5b22896f4a6fa14fa9510f294d87 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:05:27.675709 22168 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e0c5b22896f4a6fa14fa9510f294d87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e0c5b22896f4a6fa14fa9510f294d87" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 33827 } }
I20260806 03:05:27.685609 22169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e0c5b22896f4a6fa14fa9510f294d87 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e0c5b22896f4a6fa14fa9510f294d87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e0c5b22896f4a6fa14fa9510f294d87" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 33827 } } }
I20260806 03:05:27.686139 22169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e0c5b22896f4a6fa14fa9510f294d87 [sys.catalog]: This master's current role is: LEADER
I20260806 03:05:27.686898 22171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e0c5b22896f4a6fa14fa9510f294d87 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e0c5b22896f4a6fa14fa9510f294d87. Latest consensus state: current_term: 1 leader_uuid: "6e0c5b22896f4a6fa14fa9510f294d87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e0c5b22896f4a6fa14fa9510f294d87" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 33827 } } }
I20260806 03:05:27.687243 22171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e0c5b22896f4a6fa14fa9510f294d87 [sys.catalog]: This master's current role is: LEADER
I20260806 03:05:27.698381 22182 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:05:27.705457 22182 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:05:27.705976 22166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:27.680951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53851 (local address 127.31.74.254:33827)
0806 03:05:27.681221 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:27.681232 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:27.681894 (+ 662us) server_negotiation.cc:407] Connection header received
0806 03:05:27.682113 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:27.682128 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:27.682212 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:27.682287 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:27.684159 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:27.685706 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:27.688593 (+ 2887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:27.689701 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:27.689861 (+ 160us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:05:27.695375 (+ 5514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:05:27.695388 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:05:27.695396 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:05:27.695460 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:05:27.700137 (+ 4677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:05:27.701599 (+ 1462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:05:27.701609 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:05:27.701612 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:05:27.702004 (+ 392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:05:27.702932 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:05:27.702939 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:05:27.702942 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:05:27.703474 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:05:27.703757 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0806 03:05:27.704988 (+ 1231us) server_negotiation.cc:299] Negotiation successful
0806 03:05:27.705375 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 03:05:27.716881 22182 catalog_manager.cc:1347] Generated new cluster ID: cfe0df4d754e4e8eb05bd4710d65eff5
I20260806 03:05:27.717027 22182 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:05:27.749135 22182 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:05:27.750524 22182 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:05:27.758879 22182 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e0c5b22896f4a6fa14fa9510f294d87: Generated new TSK 0
I20260806 03:05:27.759770 22182 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 03:05:27.766636 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 22095
I20260806 03:05:27.768177 22158 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 03:05:27.875747 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 22095
W20260806 03:05:27.900707 22080 connection.cc:535] client connection to 127.31.74.254:33827 recv error: Network error: failed to read from TLS socket (remote: 127.31.74.254:33827): Connection reset by peer (error 104)
2026-08-06T03:05:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2477 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785985105255475-32043-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 03:05:28 dist-test-slave-dhph krb5kdc[22196](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 03:05:28 dist-test-slave-dhph krb5kdc[22196](info): set up 2 sockets
Aug 06 03:05:28 dist-test-slave-dhph krb5kdc[22196](info): commencing operation
krb5kdc: starting...
W20260806 03:05:29.137221 32043 mini_kdc.cc:121] Time spent starting KDC: real 1.215s user 0.005s sys 0.055s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785985105255475-32043-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785985105255475-32043-0/krb5kdc/test-user.keytab.
Aug 06 03:05:29 dist-test-slave-dhph krb5kdc[22196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985529, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T03:05:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:05:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.74.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab.
Entry for principal kudu/127.31.74.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab.
I20260806 03:05:29.552835 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785985105255475-32043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785985105255475-32043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785985105255475-32043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:45949
--webserver_interface=127.31.74.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab
--principal=kudu/127.31.74.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:44909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.74.254:45949
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:05:29.954432 22210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:05:29.954788 22210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:05:29.954934 22210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:05:29.963388 22210 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:05:29.963559 22210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:05:29.963653 22210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:05:29.963773 22210 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:05:29.963850 22210 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:05:29.975648 22210 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:44909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785985105255475-32043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785985105255475-32043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.74.254:45949
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab
--principal=kudu/127.31.74.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:45949
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.74.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785985105255475-32043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:05:29.977144 22210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:05:29.978734 22210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:05:29.998981 22215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:05:29.999117 22216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:05:29.999117 22218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:05:30.000733 22210 server_base.cc:1034] running on GCE node
I20260806 03:05:30.001696 22210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:05:30.003783 22210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:05:30.005378 22210 hybrid_clock.cc:648] HybridClock initialized: now 1785985530005224 us; error 67 us; skew 500 ppm
I20260806 03:05:30.006134 22210 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 03:05:30 dist-test-slave-dhph krb5kdc[22196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985530, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:05:30.016158 22210 init.cc:377] Logged in from keytab as kudu/127.31.74.254@KRBTEST.COM (short username kudu)
I20260806 03:05:30.018271 22210 webserver.cc:460] Webserver started at http://127.31.74.254:38511/ using document root <none> and password file <none>
I20260806 03:05:30.019158 22210 fs_manager.cc:359] Metadata directory not provided
I20260806 03:05:30.019335 22210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:05:30.019694 22210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:05:30.023154 22210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785985105255475-32043-0/minicluster-data/master-0/data/instance:
uuid: "f3d00cddbfb54121b576e6ff756fa520"
format_stamp: "Formatted at 2026-08-06 03:05:30 on dist-test-slave-dhph"
server_key: "823f8ac0b99846c04feb9a9423cc2623"
server_key_iv: "c14a70b4bb4b2253efae179cf3a7d3dd"
server_key_version: "encryptionkey@0"
I20260806 03:05:30.024045 22210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785985105255475-32043-0/minicluster-data/master-0/wal/instance:
uuid: "f3d00cddbfb54121b576e6ff756fa520"
format_stamp: "Formatted at 2026-08-06 03:05:30 on dist-test-slave-dhph"
server_key: "823f8ac0b99846c04feb9a9423cc2623"
server_key_iv: "c14a70b4bb4b2253efae179cf3a7d3dd"
server_key_version: "encryptionkey@0"
I20260806 03:05:30.030306 22210 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.004s
I20260806 03:05:30.035012 22225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:05:30.036962 22210 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 03:05:30.037197 22210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785985105255475-32043-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785985105255475-32043-0/minicluster-data/master-0/wal
uuid: "f3d00cddbfb54121b576e6ff756fa520"
format_stamp: "Formatted at 2026-08-06 03:05:30 on dist-test-slave-dhph"
server_key: "823f8ac0b99846c04feb9a9423cc2623"
server_key_iv: "c14a70b4bb4b2253efae179cf3a7d3dd"
server_key_version: "encryptionkey@0"
I20260806 03:05:30.037464 22210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785985105255475-32043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785985105255475-32043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785985105255475-32043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:05:30.053622 22210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:05:30.058449 22210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:05:30.058836 22210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:05:30.081192 22210 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.254:45949
I20260806 03:05:30.081223 22278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.254:45949 every 8 connection(s)
I20260806 03:05:30.082814 22210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
I20260806 03:05:30.087138 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 22210
I20260806 03:05:30.087245 22280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:05:30.087361 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785985105255475-32043-0/minicluster-data/master-0/wal/instance
I20260806 03:05:30.088014 32043 external_mini_cluster.cc:1428] Setting key a815a0ea93b26cea65c1b0be09e60c09
I20260806 03:05:30.102849 22280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3d00cddbfb54121b576e6ff756fa520: Bootstrap starting.
I20260806 03:05:30.103807 22284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:30.090113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38387 (local address 127.31.74.254:45949)
0806 03:05:30.091123 (+ 1010us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:30.091135 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:30.091177 (+ 42us) server_negotiation.cc:407] Connection header received
0806 03:05:30.093096 (+ 1919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:30.093121 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:30.093882 (+ 761us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:30.094239 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:30.095009 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:30.096323 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:30.098215 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:30.099160 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:30.099314 (+ 154us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:05:30.100791 (+ 1477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:05:30.100801 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:05:30.100817 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:05:30.101472 (+ 655us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:05:30.101560 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:05:30.101590 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:05:30.101734 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0806 03:05:30.102292 (+ 558us) server_negotiation.cc:299] Negotiation successful
0806 03:05:30.102485 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":770,"thread_start_us":517,"threads_started":1}
I20260806 03:05:30.106506 22280 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f3d00cddbfb54121b576e6ff756fa520: Neither blocks nor log segments found. Creating new log.
I20260806 03:05:30.107878 22280 log.cc:824] T 00000000000000000000000000000000 P f3d00cddbfb54121b576e6ff756fa520: Log is configured to *not* fsync() on all Append() calls
I20260806 03:05:30.108412 32043 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 03:05:30.111346 22280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3d00cddbfb54121b576e6ff756fa520: No bootstrap required, opened a new log
I20260806 03:05:30.124300 22284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:30.114849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36093 (local address 127.31.74.254:45949)
0806 03:05:30.115116 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:30.115126 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:30.115668 (+ 542us) server_negotiation.cc:407] Connection header received
0806 03:05:30.115893 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:30.115897 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:30.115983 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:30.116096 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:30.116774 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:30.117594 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:30.120468 (+ 2874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:30.121076 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:30.121330 (+ 254us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:05:30.123072 (+ 1742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:05:30.123080 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:05:30.123088 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:05:30.123123 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:05:30.123224 (+ 101us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:05:30.123248 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:05:30.123426 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0806 03:05:30.123930 (+ 504us) server_negotiation.cc:299] Negotiation successful
0806 03:05:30.124007 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260806 03:05:30.124886 22280 raft_consensus.cc:348] T 00000000000000000000000000000000 P f3d00cddbfb54121b576e6ff756fa520 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3d00cddbfb54121b576e6ff756fa520" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 45949 } }
I20260806 03:05:30.125427 22280 raft_consensus.cc:374] T 00000000000000000000000000000000 P f3d00cddbfb54121b576e6ff756fa520 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:05:30.125540 22280 raft_consensus.cc:729] T 00000000000000000000000000000000 P f3d00cddbfb54121b576e6ff756fa520 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3d00cddbfb54121b576e6ff756fa520, State: Initialized, Role: FOLLOWER
I20260806 03:05:30.126150 22280 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3d00cddbfb54121b576e6ff756fa520 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3d00cddbfb54121b576e6ff756fa520" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 45949 } }
I20260806 03:05:30.126385 22280 raft_consensus.cc:388] T 00000000000000000000000000000000 P f3d00cddbfb54121b576e6ff756fa520 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:05:30.126493 22280 raft_consensus.cc:482] T 00000000000000000000000000000000 P f3d00cddbfb54121b576e6ff756fa520 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:05:30.126727 22280 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f3d00cddbfb54121b576e6ff756fa520 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:05:30.129101 22280 raft_consensus.cc:504] T 00000000000000000000000000000000 P f3d00cddbfb54121b576e6ff756fa520 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3d00cddbfb54121b576e6ff756fa520" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 45949 } }
I20260806 03:05:30.129835 22280 leader_election.cc:302] T 00000000000000000000000000000000 P f3d00cddbfb54121b576e6ff756fa520 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f3d00cddbfb54121b576e6ff756fa520; no voters:
I20260806 03:05:30.130947 22280 leader_election.cc:288] T 00000000000000000000000000000000 P f3d00cddbfb54121b576e6ff756fa520 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:05:30.131058 22291 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f3d00cddbfb54121b576e6ff756fa520 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:05:30.132335 22291 raft_consensus.cc:686] T 00000000000000000000000000000000 P f3d00cddbfb54121b576e6ff756fa520 [term 1 LEADER]: Becoming Leader. State: Replica: f3d00cddbfb54121b576e6ff756fa520, State: Running, Role: LEADER
I20260806 03:05:30.133374 22291 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3d00cddbfb54121b576e6ff756fa520 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3d00cddbfb54121b576e6ff756fa520" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 45949 } }
I20260806 03:05:30.134732 22280 sys_catalog.cc:564] T 00000000000000000000000000000000 P f3d00cddbfb54121b576e6ff756fa520 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:05:30.137135 22292 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3d00cddbfb54121b576e6ff756fa520 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f3d00cddbfb54121b576e6ff756fa520" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3d00cddbfb54121b576e6ff756fa520" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 45949 } } }
I20260806 03:05:30.137123 22293 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3d00cddbfb54121b576e6ff756fa520 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3d00cddbfb54121b576e6ff756fa520. Latest consensus state: current_term: 1 leader_uuid: "f3d00cddbfb54121b576e6ff756fa520" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3d00cddbfb54121b576e6ff756fa520" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 45949 } } }
I20260806 03:05:30.138247 22292 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3d00cddbfb54121b576e6ff756fa520 [sys.catalog]: This master's current role is: LEADER
I20260806 03:05:30.138262 22293 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3d00cddbfb54121b576e6ff756fa520 [sys.catalog]: This master's current role is: LEADER
I20260806 03:05:30.142292 22296 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:05:30.149776 22296 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:05:30.165480 22296 catalog_manager.cc:1347] Generated new cluster ID: e637564c30a2444fa2d6ffae41d511a9
I20260806 03:05:30.165634 22296 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:05:30.180724 22296 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:05:30.182402 22296 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:05:30.201359 22296 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f3d00cddbfb54121b576e6ff756fa520: Generated new TSK 0
I20260806 03:05:30.202422 22296 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 03:05:30.212723 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 22210
W20260806 03:05:30.214107 22274 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:38387
W20260806 03:05:30.215250 32043 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 22210: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 03:05:30.215466 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 22210
W20260806 03:05:30.241981 22194 connection.cc:535] client connection to 127.31.74.254:45949 recv error: Network error: failed to read from TLS socket (remote: 127.31.74.254:45949): Connection reset by peer (error 104)
2026-08-06T03:05:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2340 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785985105255475-32043-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 03:05:30 dist-test-slave-dhph krb5kdc[22313](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 03:05:30 dist-test-slave-dhph krb5kdc[22313](info): set up 2 sockets
Aug 06 03:05:30 dist-test-slave-dhph krb5kdc[22313](info): commencing operation
krb5kdc: starting...
W20260806 03:05:31.492066 32043 mini_kdc.cc:121] Time spent starting KDC: real 1.230s user 0.000s sys 0.058s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785985105255475-32043-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785985105255475-32043-0/krb5kdc/test-user.keytab.
Aug 06 03:05:31 dist-test-slave-dhph krb5kdc[22313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985531, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T03:05:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:05:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.74.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab.
Entry for principal kudu/127.31.74.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab.
I20260806 03:05:31.903100 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785985105255475-32043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785985105255475-32043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785985105255475-32043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:39269
--webserver_interface=127.31.74.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab
--principal=kudu/127.31.74.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:45385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.74.254:39269
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:05:32.293357 22327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:05:32.293733 22327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:05:32.293918 22327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:05:32.301532 22327 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:05:32.301667 22327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:05:32.301770 22327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:05:32.301891 22327 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:05:32.301976 22327 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:05:32.313470 22327 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:45385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785985105255475-32043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785985105255475-32043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.74.254:39269
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab
--principal=kudu/127.31.74.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:39269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.74.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785985105255475-32043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:05:32.315037 22327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:05:32.316543 22327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:05:32.331050 22335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:05:32.331125 22332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:05:32.331580 22333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:05:32.332357 22327 server_base.cc:1034] running on GCE node
I20260806 03:05:32.333494 22327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:05:32.335286 22327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:05:32.336570 22327 hybrid_clock.cc:648] HybridClock initialized: now 1785985532336539 us; error 51 us; skew 500 ppm
I20260806 03:05:32.336978 22327 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 03:05:32 dist-test-slave-dhph krb5kdc[22313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985532, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:05:32.340631 22327 init.cc:377] Logged in from keytab as kudu/127.31.74.254@KRBTEST.COM (short username kudu)
I20260806 03:05:32.342500 22327 webserver.cc:460] Webserver started at http://127.31.74.254:41457/ using document root <none> and password file <none>
I20260806 03:05:32.343302 22327 fs_manager.cc:359] Metadata directory not provided
I20260806 03:05:32.343524 22327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:05:32.343984 22327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:05:32.347368 22327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785985105255475-32043-0/minicluster-data/master-0/data/instance:
uuid: "946dd814ccff424b8138b3d6630d8aba"
format_stamp: "Formatted at 2026-08-06 03:05:32 on dist-test-slave-dhph"
server_key: "fe03e069eb1de108dce958fad2d12b76"
server_key_iv: "899a092d7dd4954be35a5bcce0f55985"
server_key_version: "encryptionkey@0"
I20260806 03:05:32.348330 22327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785985105255475-32043-0/minicluster-data/master-0/wal/instance:
uuid: "946dd814ccff424b8138b3d6630d8aba"
format_stamp: "Formatted at 2026-08-06 03:05:32 on dist-test-slave-dhph"
server_key: "fe03e069eb1de108dce958fad2d12b76"
server_key_iv: "899a092d7dd4954be35a5bcce0f55985"
server_key_version: "encryptionkey@0"
I20260806 03:05:32.354311 22327 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260806 03:05:32.358547 22342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:05:32.360236 22327 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 03:05:32.360466 22327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785985105255475-32043-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785985105255475-32043-0/minicluster-data/master-0/wal
uuid: "946dd814ccff424b8138b3d6630d8aba"
format_stamp: "Formatted at 2026-08-06 03:05:32 on dist-test-slave-dhph"
server_key: "fe03e069eb1de108dce958fad2d12b76"
server_key_iv: "899a092d7dd4954be35a5bcce0f55985"
server_key_version: "encryptionkey@0"
I20260806 03:05:32.360711 22327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785985105255475-32043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785985105255475-32043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785985105255475-32043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:05:32.367908 22327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:05:32.373272 22327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:05:32.373649 22327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:05:32.396060 22327 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.254:39269
I20260806 03:05:32.396154 22395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.254:39269 every 8 connection(s)
I20260806 03:05:32.397559 22327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
I20260806 03:05:32.401453 22396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:05:32.405380 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 22327
I20260806 03:05:32.405561 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785985105255475-32043-0/minicluster-data/master-0/wal/instance
I20260806 03:05:32.406136 32043 external_mini_cluster.cc:1428] Setting key d429ca43c137cb22f6c372d0f8fb015c
I20260806 03:05:32.415635 22399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:32.408178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48065 (local address 127.31.74.254:39269)
0806 03:05:32.409120 (+ 942us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:32.409132 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:32.409184 (+ 52us) server_negotiation.cc:407] Connection header received
0806 03:05:32.411105 (+ 1921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:32.411131 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:32.411532 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:32.411907 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:32.413317 (+ 1410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:05:32.413324 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:05:32.413337 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:05:32.413358 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:05:32.413448 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:05:32.413471 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:05:32.413728 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0806 03:05:32.413844 (+ 116us) server_negotiation.cc:299] Negotiation successful
0806 03:05:32.414125 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":714,"thread_start_us":459,"threads_started":1}
I20260806 03:05:32.416121 22396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 946dd814ccff424b8138b3d6630d8aba: Bootstrap starting.
I20260806 03:05:32.419497 22396 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 946dd814ccff424b8138b3d6630d8aba: Neither blocks nor log segments found. Creating new log.
I20260806 03:05:32.420200 32043 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 03:05:32.420828 22396 log.cc:824] T 00000000000000000000000000000000 P 946dd814ccff424b8138b3d6630d8aba: Log is configured to *not* fsync() on all Append() calls
I20260806 03:05:32.424686 22396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 946dd814ccff424b8138b3d6630d8aba: No bootstrap required, opened a new log
I20260806 03:05:32.429201 22399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:32.425960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60869 (local address 127.31.74.254:39269)
0806 03:05:32.426170 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:32.426178 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:32.426665 (+ 487us) server_negotiation.cc:407] Connection header received
0806 03:05:32.427001 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:32.427006 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:32.427073 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:32.427199 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:32.428249 (+ 1050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:05:32.428256 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:05:32.428260 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:05:32.428279 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:05:32.428334 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:05:32.428355 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:05:32.428496 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0806 03:05:32.428799 (+ 303us) server_negotiation.cc:299] Negotiation successful
0806 03:05:32.428884 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 03:05:32.435151 22396 raft_consensus.cc:348] T 00000000000000000000000000000000 P 946dd814ccff424b8138b3d6630d8aba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "946dd814ccff424b8138b3d6630d8aba" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 39269 } }
I20260806 03:05:32.435583 22396 raft_consensus.cc:374] T 00000000000000000000000000000000 P 946dd814ccff424b8138b3d6630d8aba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:05:32.435689 22396 raft_consensus.cc:729] T 00000000000000000000000000000000 P 946dd814ccff424b8138b3d6630d8aba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 946dd814ccff424b8138b3d6630d8aba, State: Initialized, Role: FOLLOWER
I20260806 03:05:32.436311 22396 consensus_queue.cc:260] T 00000000000000000000000000000000 P 946dd814ccff424b8138b3d6630d8aba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "946dd814ccff424b8138b3d6630d8aba" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 39269 } }
I20260806 03:05:32.436546 22396 raft_consensus.cc:388] T 00000000000000000000000000000000 P 946dd814ccff424b8138b3d6630d8aba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:05:32.436741 22396 raft_consensus.cc:482] T 00000000000000000000000000000000 P 946dd814ccff424b8138b3d6630d8aba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:05:32.436919 22396 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 946dd814ccff424b8138b3d6630d8aba [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:05:32.439865 22396 raft_consensus.cc:504] T 00000000000000000000000000000000 P 946dd814ccff424b8138b3d6630d8aba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "946dd814ccff424b8138b3d6630d8aba" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 39269 } }
I20260806 03:05:32.440477 22396 leader_election.cc:302] T 00000000000000000000000000000000 P 946dd814ccff424b8138b3d6630d8aba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 946dd814ccff424b8138b3d6630d8aba; no voters:
I20260806 03:05:32.441254 22396 leader_election.cc:288] T 00000000000000000000000000000000 P 946dd814ccff424b8138b3d6630d8aba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:05:32.441475 22406 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 946dd814ccff424b8138b3d6630d8aba [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:05:32.442193 22406 raft_consensus.cc:686] T 00000000000000000000000000000000 P 946dd814ccff424b8138b3d6630d8aba [term 1 LEADER]: Becoming Leader. State: Replica: 946dd814ccff424b8138b3d6630d8aba, State: Running, Role: LEADER
I20260806 03:05:32.442878 22406 consensus_queue.cc:237] T 00000000000000000000000000000000 P 946dd814ccff424b8138b3d6630d8aba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "946dd814ccff424b8138b3d6630d8aba" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 39269 } }
I20260806 03:05:32.443717 22396 sys_catalog.cc:564] T 00000000000000000000000000000000 P 946dd814ccff424b8138b3d6630d8aba [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:05:32.446161 22408 sys_catalog.cc:455] T 00000000000000000000000000000000 P 946dd814ccff424b8138b3d6630d8aba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 946dd814ccff424b8138b3d6630d8aba. Latest consensus state: current_term: 1 leader_uuid: "946dd814ccff424b8138b3d6630d8aba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "946dd814ccff424b8138b3d6630d8aba" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 39269 } } }
I20260806 03:05:32.446681 22408 sys_catalog.cc:458] T 00000000000000000000000000000000 P 946dd814ccff424b8138b3d6630d8aba [sys.catalog]: This master's current role is: LEADER
I20260806 03:05:32.446102 22407 sys_catalog.cc:455] T 00000000000000000000000000000000 P 946dd814ccff424b8138b3d6630d8aba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "946dd814ccff424b8138b3d6630d8aba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "946dd814ccff424b8138b3d6630d8aba" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 39269 } } }
I20260806 03:05:32.447942 22407 sys_catalog.cc:458] T 00000000000000000000000000000000 P 946dd814ccff424b8138b3d6630d8aba [sys.catalog]: This master's current role is: LEADER
I20260806 03:05:32.450505 22414 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:05:32.456561 22414 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:05:32.471405 22414 catalog_manager.cc:1347] Generated new cluster ID: d41274ceddc142b0905f694bc3e2a870
I20260806 03:05:32.471553 22414 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:05:32.490883 22414 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:05:32.491513 22414 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:05:32.515833 22414 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 946dd814ccff424b8138b3d6630d8aba: Generated new TSK 0
I20260806 03:05:32.516768 22414 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 03:05:32.528412 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 22327
W20260806 03:05:32.529706 22390 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:48065
W20260806 03:05:32.530627 32043 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 22327: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 03:05:32.530809 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 22327
2026-08-06T03:05:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (2309 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785985105255475-32043-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 03:05:32 dist-test-slave-dhph krb5kdc[22429](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 03:05:32 dist-test-slave-dhph krb5kdc[22429](info): set up 2 sockets
Aug 06 03:05:32 dist-test-slave-dhph krb5kdc[22429](info): commencing operation
krb5kdc: starting...
W20260806 03:05:33.834661 32043 mini_kdc.cc:121] Time spent starting KDC: real 1.262s user 0.000s sys 0.057s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785985105255475-32043-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785985105255475-32043-0/krb5kdc/test-user.keytab.
Aug 06 03:05:34 dist-test-slave-dhph krb5kdc[22429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985534, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T03:05:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:05:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.74.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab.
Entry for principal kudu/127.31.74.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab.
I20260806 03:05:34.248327 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785985105255475-32043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785985105255475-32043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785985105255475-32043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:46135
--webserver_interface=127.31.74.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab
--principal=kudu/127.31.74.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:37197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.74.254:46135
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:05:34.681811 22442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:05:34.682174 22442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:05:34.682372 22442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:05:34.690891 22442 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:05:34.691033 22442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:05:34.691082 22442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:05:34.691139 22442 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:05:34.691185 22442 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:05:34.704317 22442 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:37197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785985105255475-32043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785985105255475-32043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.74.254:46135
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab
--principal=kudu/127.31.74.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:46135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.74.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785985105255475-32043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:05:34.705677 22442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:05:34.707511 22442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:05:34.722512 22447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:05:34.722450 22448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:05:34.722546 22450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:05:34.723201 22442 server_base.cc:1034] running on GCE node
I20260806 03:05:34.724344 22442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:05:34.725958 22442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:05:34.727210 22442 hybrid_clock.cc:648] HybridClock initialized: now 1785985534727147 us; error 41 us; skew 500 ppm
I20260806 03:05:34.727711 22442 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 03:05:34 dist-test-slave-dhph krb5kdc[22429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985534, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:05:34.731680 22442 init.cc:377] Logged in from keytab as kudu/127.31.74.254@KRBTEST.COM (short username kudu)
I20260806 03:05:34.733618 22442 webserver.cc:460] Webserver started at http://127.31.74.254:38347/ using document root <none> and password file <none>
I20260806 03:05:34.734480 22442 fs_manager.cc:359] Metadata directory not provided
I20260806 03:05:34.734685 22442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:05:34.735075 22442 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:05:34.739508 22442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785985105255475-32043-0/minicluster-data/master-0/data/instance:
uuid: "7a5bf67e32274f5899cad6b7425377cf"
format_stamp: "Formatted at 2026-08-06 03:05:34 on dist-test-slave-dhph"
server_key: "9e733ac42ffeb0085246e8d672c604aa"
server_key_iv: "36f5d96763cbae29e86eaba9b7e070c9"
server_key_version: "encryptionkey@0"
I20260806 03:05:34.740499 22442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785985105255475-32043-0/minicluster-data/master-0/wal/instance:
uuid: "7a5bf67e32274f5899cad6b7425377cf"
format_stamp: "Formatted at 2026-08-06 03:05:34 on dist-test-slave-dhph"
server_key: "9e733ac42ffeb0085246e8d672c604aa"
server_key_iv: "36f5d96763cbae29e86eaba9b7e070c9"
server_key_version: "encryptionkey@0"
I20260806 03:05:34.747612 22442 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.004s
I20260806 03:05:34.752714 22457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:05:34.755592 22442 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260806 03:05:34.755906 22442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785985105255475-32043-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785985105255475-32043-0/minicluster-data/master-0/wal
uuid: "7a5bf67e32274f5899cad6b7425377cf"
format_stamp: "Formatted at 2026-08-06 03:05:34 on dist-test-slave-dhph"
server_key: "9e733ac42ffeb0085246e8d672c604aa"
server_key_iv: "36f5d96763cbae29e86eaba9b7e070c9"
server_key_version: "encryptionkey@0"
I20260806 03:05:34.756340 22442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785985105255475-32043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785985105255475-32043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785985105255475-32043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:05:34.808897 22442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:05:34.814864 22442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:05:34.815200 22442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:05:34.841969 22442 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.254:46135
I20260806 03:05:34.842028 22511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.254:46135 every 8 connection(s)
I20260806 03:05:34.844192 22442 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
I20260806 03:05:34.846992 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 22442
I20260806 03:05:34.847220 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785985105255475-32043-0/minicluster-data/master-0/wal/instance
I20260806 03:05:34.847802 32043 external_mini_cluster.cc:1428] Setting key b45910ee05d49a22786cc2fc58ec2e80
I20260806 03:05:34.849792 22512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:05:34.864729 22512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a5bf67e32274f5899cad6b7425377cf: Bootstrap starting.
Aug 06 03:05:34 dist-test-slave-dhph krb5kdc[22429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985534, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 03:05:34.869320 22512 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a5bf67e32274f5899cad6b7425377cf: Neither blocks nor log segments found. Creating new log.
I20260806 03:05:34.870817 22512 log.cc:824] T 00000000000000000000000000000000 P 7a5bf67e32274f5899cad6b7425377cf: Log is configured to *not* fsync() on all Append() calls
I20260806 03:05:34.874919 22512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a5bf67e32274f5899cad6b7425377cf: No bootstrap required, opened a new log
I20260806 03:05:34.876436 22515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:34.849948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43833 (local address 127.31.74.254:46135)
0806 03:05:34.850844 (+ 896us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:34.850856 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:34.850897 (+ 41us) server_negotiation.cc:407] Connection header received
0806 03:05:34.852548 (+ 1651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:34.852569 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:34.852994 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:34.853345 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:34.855494 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:34.857957 (+ 2463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:34.860010 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:34.861160 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:34.861410 (+ 250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:05:34.867199 (+ 5789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:05:34.867208 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:05:34.867225 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:05:34.867288 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:05:34.871259 (+ 3971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:05:34.872301 (+ 1042us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:05:34.872309 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:05:34.872316 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:05:34.872526 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:05:34.872978 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:05:34.872983 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:05:34.872985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:05:34.873763 (+ 778us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:05:34.874144 (+ 381us) server_negotiation.cc:1035] Waiting for connection context
0806 03:05:34.874779 (+ 635us) server_negotiation.cc:299] Negotiation successful
0806 03:05:34.875319 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":667,"thread_start_us":394,"threads_started":1}
I20260806 03:05:34.881148 32043 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 03:05:34.885684 22512 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a5bf67e32274f5899cad6b7425377cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a5bf67e32274f5899cad6b7425377cf" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 46135 } }
I20260806 03:05:34.886206 22512 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a5bf67e32274f5899cad6b7425377cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:05:34.886365 22512 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a5bf67e32274f5899cad6b7425377cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a5bf67e32274f5899cad6b7425377cf, State: Initialized, Role: FOLLOWER
I20260806 03:05:34.887377 22512 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a5bf67e32274f5899cad6b7425377cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a5bf67e32274f5899cad6b7425377cf" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 46135 } }
I20260806 03:05:34.887943 22512 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a5bf67e32274f5899cad6b7425377cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:05:34.888136 22512 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a5bf67e32274f5899cad6b7425377cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:05:34.888398 22512 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a5bf67e32274f5899cad6b7425377cf [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:05:34.892771 22512 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a5bf67e32274f5899cad6b7425377cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a5bf67e32274f5899cad6b7425377cf" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 46135 } }
I20260806 03:05:34.894102 22512 leader_election.cc:302] T 00000000000000000000000000000000 P 7a5bf67e32274f5899cad6b7425377cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a5bf67e32274f5899cad6b7425377cf; no voters:
I20260806 03:05:34.895450 22512 leader_election.cc:288] T 00000000000000000000000000000000 P 7a5bf67e32274f5899cad6b7425377cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:05:34.895612 22522 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a5bf67e32274f5899cad6b7425377cf [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:05:34.896641 22522 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a5bf67e32274f5899cad6b7425377cf [term 1 LEADER]: Becoming Leader. State: Replica: 7a5bf67e32274f5899cad6b7425377cf, State: Running, Role: LEADER
I20260806 03:05:34.897416 22522 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a5bf67e32274f5899cad6b7425377cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a5bf67e32274f5899cad6b7425377cf" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 46135 } }
I20260806 03:05:34.899138 22512 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a5bf67e32274f5899cad6b7425377cf [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:05:34.900640 22522 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a5bf67e32274f5899cad6b7425377cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a5bf67e32274f5899cad6b7425377cf. Latest consensus state: current_term: 1 leader_uuid: "7a5bf67e32274f5899cad6b7425377cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a5bf67e32274f5899cad6b7425377cf" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 46135 } } }
I20260806 03:05:34.900640 22524 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a5bf67e32274f5899cad6b7425377cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a5bf67e32274f5899cad6b7425377cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a5bf67e32274f5899cad6b7425377cf" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 46135 } } }
I20260806 03:05:34.901194 22524 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a5bf67e32274f5899cad6b7425377cf [sys.catalog]: This master's current role is: LEADER
I20260806 03:05:34.901050 22522 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a5bf67e32274f5899cad6b7425377cf [sys.catalog]: This master's current role is: LEADER
I20260806 03:05:34.908490 22527 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:05:34.913587 22515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:34.887480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46341 (local address 127.31.74.254:46135)
0806 03:05:34.887768 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:34.887779 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:34.888352 (+ 573us) server_negotiation.cc:407] Connection header received
0806 03:05:34.888556 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:34.888564 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:34.888678 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:34.888932 (+ 254us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:34.890867 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:34.892174 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:34.893994 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:34.895422 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:34.895621 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:05:34.900036 (+ 4415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:05:34.900048 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:05:34.900055 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:05:34.900126 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:05:34.906624 (+ 6498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:05:34.908085 (+ 1461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:05:34.908095 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:05:34.908102 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:05:34.908425 (+ 323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:05:34.909147 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:05:34.909157 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:05:34.909163 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:05:34.909729 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:05:34.910196 (+ 467us) server_negotiation.cc:1035] Waiting for connection context
0806 03:05:34.912159 (+ 1963us) server_negotiation.cc:299] Negotiation successful
0806 03:05:34.912931 (+ 772us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 03:05:34.917640 22527 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:05:34.937716 22527 catalog_manager.cc:1347] Generated new cluster ID: 1293035a097f4698b3e55f075c780e6d
I20260806 03:05:34.937860 22527 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:05:34.971747 22527 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:05:34.974202 22527 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:05:34.989042 22527 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a5bf67e32274f5899cad6b7425377cf: Generated new TSK 0
I20260806 03:05:34.990417 22527 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 03:05:35.009125 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 22442
I20260806 03:05:35.011031 22507 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 03:05:35.135552 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 22442
2026-08-06T03:05:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2607 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785985105255475-32043-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 03:05:35 dist-test-slave-dhph krb5kdc[22544](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 03:05:35 dist-test-slave-dhph krb5kdc[22544](info): set up 2 sockets
Aug 06 03:05:35 dist-test-slave-dhph krb5kdc[22544](info): commencing operation
krb5kdc: starting...
W20260806 03:05:36.414786 32043 mini_kdc.cc:121] Time spent starting KDC: real 1.236s user 0.002s sys 0.062s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785985105255475-32043-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785985105255475-32043-0/krb5kdc/test-user.keytab.
Aug 06 03:05:36 dist-test-slave-dhph krb5kdc[22544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985536, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T03:05:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:05:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.74.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab.
Entry for principal kudu/127.31.74.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab.
I20260806 03:05:36.832705 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785985105255475-32043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785985105255475-32043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785985105255475-32043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:43771
--webserver_interface=127.31.74.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab
--principal=kudu/127.31.74.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:41571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.74.254:43771
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:05:37.257287 22558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:05:37.257633 22558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:05:37.257799 22558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:05:37.267856 22558 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:05:37.268002 22558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:05:37.268116 22558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:05:37.268213 22558 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:05:37.268257 22558 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:05:37.280313 22558 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:41571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785985105255475-32043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785985105255475-32043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.74.254:43771
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab
--principal=kudu/127.31.74.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:43771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.74.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785985105255475-32043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:05:37.281778 22558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:05:37.283172 22558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:05:37.296468 22563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:05:37.296397 22564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:05:37.296499 22566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:05:37.297475 22558 server_base.cc:1034] running on GCE node
I20260806 03:05:37.298650 22558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:05:37.300421 22558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:05:37.301705 22558 hybrid_clock.cc:648] HybridClock initialized: now 1785985537301666 us; error 57 us; skew 500 ppm
I20260806 03:05:37.302135 22558 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 03:05:37 dist-test-slave-dhph krb5kdc[22544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985537, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:05:37.305810 22558 init.cc:377] Logged in from keytab as kudu/127.31.74.254@KRBTEST.COM (short username kudu)
I20260806 03:05:37.307726 22558 webserver.cc:460] Webserver started at http://127.31.74.254:33909/ using document root <none> and password file <none>
I20260806 03:05:37.308471 22558 fs_manager.cc:359] Metadata directory not provided
I20260806 03:05:37.308627 22558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:05:37.308952 22558 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:05:37.312261 22558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785985105255475-32043-0/minicluster-data/master-0/data/instance:
uuid: "65e2590c8d9e4c01a52811325c21bff8"
format_stamp: "Formatted at 2026-08-06 03:05:37 on dist-test-slave-dhph"
server_key: "01c91db870aca7743f709f5a495c0cce"
server_key_iv: "1b7e6700ec14593ff49e194cd884fda5"
server_key_version: "encryptionkey@0"
I20260806 03:05:37.313108 22558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785985105255475-32043-0/minicluster-data/master-0/wal/instance:
uuid: "65e2590c8d9e4c01a52811325c21bff8"
format_stamp: "Formatted at 2026-08-06 03:05:37 on dist-test-slave-dhph"
server_key: "01c91db870aca7743f709f5a495c0cce"
server_key_iv: "1b7e6700ec14593ff49e194cd884fda5"
server_key_version: "encryptionkey@0"
I20260806 03:05:37.319257 22558 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260806 03:05:37.324105 22573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:05:37.325974 22558 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 03:05:37.326251 22558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785985105255475-32043-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785985105255475-32043-0/minicluster-data/master-0/wal
uuid: "65e2590c8d9e4c01a52811325c21bff8"
format_stamp: "Formatted at 2026-08-06 03:05:37 on dist-test-slave-dhph"
server_key: "01c91db870aca7743f709f5a495c0cce"
server_key_iv: "1b7e6700ec14593ff49e194cd884fda5"
server_key_version: "encryptionkey@0"
I20260806 03:05:37.326740 22558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785985105255475-32043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785985105255475-32043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785985105255475-32043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:05:37.347584 22558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:05:37.354451 22558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:05:37.354904 22558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:05:37.380715 22558 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.254:43771
I20260806 03:05:37.380756 22625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.254:43771 every 8 connection(s)
I20260806 03:05:37.382386 22558 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
I20260806 03:05:37.386668 22626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:05:37.390406 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 22558
I20260806 03:05:37.390650 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785985105255475-32043-0/minicluster-data/master-0/wal/instance
I20260806 03:05:37.391170 32043 external_mini_cluster.cc:1428] Setting key 2be337925a868d5e155ab570637626e4
I20260806 03:05:37.401360 22626 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65e2590c8d9e4c01a52811325c21bff8: Bootstrap starting.
I20260806 03:05:37.406142 22626 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65e2590c8d9e4c01a52811325c21bff8: Neither blocks nor log segments found. Creating new log.
I20260806 03:05:37.407474 22629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:37.393314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33973 (local address 127.31.74.254:43771)
0806 03:05:37.394287 (+ 973us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:37.394300 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:37.394342 (+ 42us) server_negotiation.cc:407] Connection header received
0806 03:05:37.396209 (+ 1867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:37.396233 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:37.396654 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:37.397015 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:37.397743 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:37.399349 (+ 1606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:37.401383 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:37.402281 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:37.402464 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:05:37.404376 (+ 1912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:05:37.404385 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:05:37.404402 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:05:37.405155 (+ 753us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:05:37.405246 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:05:37.405276 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:05:37.405383 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 03:05:37.405698 (+ 315us) server_negotiation.cc:299] Negotiation successful
0806 03:05:37.406056 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":737,"thread_start_us":467,"threads_started":1}
I20260806 03:05:37.407470 22626 log.cc:824] T 00000000000000000000000000000000 P 65e2590c8d9e4c01a52811325c21bff8: Log is configured to *not* fsync() on all Append() calls
I20260806 03:05:37.412168 22626 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65e2590c8d9e4c01a52811325c21bff8: No bootstrap required, opened a new log
I20260806 03:05:37.413890 32043 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 03:05:37.423979 22626 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65e2590c8d9e4c01a52811325c21bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65e2590c8d9e4c01a52811325c21bff8" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 43771 } }
I20260806 03:05:37.424506 22626 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65e2590c8d9e4c01a52811325c21bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:05:37.424681 22626 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65e2590c8d9e4c01a52811325c21bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65e2590c8d9e4c01a52811325c21bff8, State: Initialized, Role: FOLLOWER
I20260806 03:05:37.425516 22626 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65e2590c8d9e4c01a52811325c21bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65e2590c8d9e4c01a52811325c21bff8" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 43771 } }
I20260806 03:05:37.425941 22626 raft_consensus.cc:388] T 00000000000000000000000000000000 P 65e2590c8d9e4c01a52811325c21bff8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:05:37.426146 22626 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65e2590c8d9e4c01a52811325c21bff8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:05:37.426414 22626 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65e2590c8d9e4c01a52811325c21bff8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:05:37.429421 22629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:37.420164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34973 (local address 127.31.74.254:43771)
0806 03:05:37.420388 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:37.420395 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:37.420882 (+ 487us) server_negotiation.cc:407] Connection header received
0806 03:05:37.421055 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:37.421061 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:37.421159 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:37.421289 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:37.422632 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:37.423921 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:37.426051 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:37.426650 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:37.426828 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:05:37.428144 (+ 1316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:05:37.428151 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:05:37.428154 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:05:37.428179 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:05:37.428225 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:05:37.428247 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:05:37.428356 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 03:05:37.428508 (+ 152us) server_negotiation.cc:299] Negotiation successful
0806 03:05:37.428619 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260806 03:05:37.431170 22626 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65e2590c8d9e4c01a52811325c21bff8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65e2590c8d9e4c01a52811325c21bff8" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 43771 } }
I20260806 03:05:37.432310 22626 leader_election.cc:302] T 00000000000000000000000000000000 P 65e2590c8d9e4c01a52811325c21bff8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65e2590c8d9e4c01a52811325c21bff8; no voters:
I20260806 03:05:37.433774 22626 leader_election.cc:288] T 00000000000000000000000000000000 P 65e2590c8d9e4c01a52811325c21bff8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:05:37.433995 22637 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65e2590c8d9e4c01a52811325c21bff8 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:05:37.435348 22637 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65e2590c8d9e4c01a52811325c21bff8 [term 1 LEADER]: Becoming Leader. State: Replica: 65e2590c8d9e4c01a52811325c21bff8, State: Running, Role: LEADER
I20260806 03:05:37.436125 22637 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65e2590c8d9e4c01a52811325c21bff8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65e2590c8d9e4c01a52811325c21bff8" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 43771 } }
I20260806 03:05:37.436578 22626 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65e2590c8d9e4c01a52811325c21bff8 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:05:37.439473 22639 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65e2590c8d9e4c01a52811325c21bff8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65e2590c8d9e4c01a52811325c21bff8. Latest consensus state: current_term: 1 leader_uuid: "65e2590c8d9e4c01a52811325c21bff8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65e2590c8d9e4c01a52811325c21bff8" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 43771 } } }
I20260806 03:05:37.440053 22639 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65e2590c8d9e4c01a52811325c21bff8 [sys.catalog]: This master's current role is: LEADER
I20260806 03:05:37.440687 22638 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65e2590c8d9e4c01a52811325c21bff8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "65e2590c8d9e4c01a52811325c21bff8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65e2590c8d9e4c01a52811325c21bff8" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 43771 } } }
I20260806 03:05:37.441119 22638 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65e2590c8d9e4c01a52811325c21bff8 [sys.catalog]: This master's current role is: LEADER
I20260806 03:05:37.444290 22645 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:05:37.452427 22645 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:05:37.463289 22645 catalog_manager.cc:1347] Generated new cluster ID: ae63af5d456d476fb18be8a1d8fafdd7
I20260806 03:05:37.463469 22645 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:05:37.491112 22645 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:05:37.493224 22645 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:05:37.510535 22645 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 65e2590c8d9e4c01a52811325c21bff8: Generated new TSK 0
I20260806 03:05:37.512224 22645 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 03:05:37.530687 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 22558
W20260806 03:05:37.531906 22621 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:33973
W20260806 03:05:37.533087 32043 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 22558: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 03:05:37.533280 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 22558
2026-08-06T03:05:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2397 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785985105255475-32043-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 03:05:37 dist-test-slave-dhph krb5kdc[22658](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 03:05:37 dist-test-slave-dhph krb5kdc[22658](info): set up 2 sockets
Aug 06 03:05:37 dist-test-slave-dhph krb5kdc[22658](info): commencing operation
krb5kdc: starting...
W20260806 03:05:38.774292 32043 mini_kdc.cc:121] Time spent starting KDC: real 1.197s user 0.003s sys 0.060s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785985105255475-32043-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785985105255475-32043-0/krb5kdc/test-user.keytab.
Aug 06 03:05:39 dist-test-slave-dhph krb5kdc[22658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T03:05:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:05:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.74.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab.
Entry for principal kudu/127.31.74.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab.
I20260806 03:05:39.197031 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785985105255475-32043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785985105255475-32043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785985105255475-32043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:36377
--webserver_interface=127.31.74.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab
--principal=kudu/127.31.74.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:36445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.74.254:36377
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:05:39.655673 22673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:05:39.656038 22673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:05:39.656147 22673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:05:39.666326 22673 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:05:39.666499 22673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:05:39.666548 22673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:05:39.666627 22673 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:05:39.666682 22673 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:05:39.678763 22673 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:36445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785985105255475-32043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785985105255475-32043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.74.254:36377
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab
--principal=kudu/127.31.74.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:36377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.74.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785985105255475-32043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:05:39.680243 22673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:05:39.681782 22673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:05:39.695850 22681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:05:39.696611 22679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:05:39.696653 22678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:05:39.697232 22673 server_base.cc:1034] running on GCE node
I20260806 03:05:39.698207 22673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:05:39.699877 22673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:05:39.701314 22673 hybrid_clock.cc:648] HybridClock initialized: now 1785985539701225 us; error 82 us; skew 500 ppm
I20260806 03:05:39.701787 22673 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 03:05:39 dist-test-slave-dhph krb5kdc[22658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985539, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:05:39.708227 22673 init.cc:377] Logged in from keytab as kudu/127.31.74.254@KRBTEST.COM (short username kudu)
I20260806 03:05:39.710153 22673 webserver.cc:460] Webserver started at http://127.31.74.254:34825/ using document root <none> and password file <none>
I20260806 03:05:39.711094 22673 fs_manager.cc:359] Metadata directory not provided
I20260806 03:05:39.711253 22673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:05:39.711650 22673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:05:39.715636 22673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785985105255475-32043-0/minicluster-data/master-0/data/instance:
uuid: "28425b83f776439587221a765fcdfa53"
format_stamp: "Formatted at 2026-08-06 03:05:39 on dist-test-slave-dhph"
server_key: "ef4dc42e400a81a6f0af7f87c386ea28"
server_key_iv: "f064bc72220453b993478b27f04f4f74"
server_key_version: "encryptionkey@0"
I20260806 03:05:39.716543 22673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785985105255475-32043-0/minicluster-data/master-0/wal/instance:
uuid: "28425b83f776439587221a765fcdfa53"
format_stamp: "Formatted at 2026-08-06 03:05:39 on dist-test-slave-dhph"
server_key: "ef4dc42e400a81a6f0af7f87c386ea28"
server_key_iv: "f064bc72220453b993478b27f04f4f74"
server_key_version: "encryptionkey@0"
I20260806 03:05:39.722784 22673 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260806 03:05:39.727654 22689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:05:39.729970 22673 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260806 03:05:39.730278 22673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785985105255475-32043-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785985105255475-32043-0/minicluster-data/master-0/wal
uuid: "28425b83f776439587221a765fcdfa53"
format_stamp: "Formatted at 2026-08-06 03:05:39 on dist-test-slave-dhph"
server_key: "ef4dc42e400a81a6f0af7f87c386ea28"
server_key_iv: "f064bc72220453b993478b27f04f4f74"
server_key_version: "encryptionkey@0"
I20260806 03:05:39.730510 22673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785985105255475-32043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785985105255475-32043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785985105255475-32043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:05:39.740140 22673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:05:39.745779 22673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:05:39.746187 22673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:05:39.772073 22673 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.254:36377
I20260806 03:05:39.772105 22742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.254:36377 every 8 connection(s)
I20260806 03:05:39.773833 22673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
I20260806 03:05:39.778282 22743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:05:39.783430 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 22673
I20260806 03:05:39.783672 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785985105255475-32043-0/minicluster-data/master-0/wal/instance
I20260806 03:05:39.784168 32043 external_mini_cluster.cc:1428] Setting key c567ee046a20ab8cda8555ade9acc002
I20260806 03:05:39.794287 22746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:39.787088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41721 (local address 127.31.74.254:36377)
0806 03:05:39.788077 (+ 989us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:39.788090 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:39.788139 (+ 49us) server_negotiation.cc:407] Connection header received
0806 03:05:39.790016 (+ 1877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:39.790043 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:39.790480 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:39.791113 (+ 633us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:39.792506 (+ 1393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:05:39.792512 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:05:39.792526 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:05:39.792541 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:05:39.792604 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:05:39.792630 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:05:39.792885 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0806 03:05:39.793066 (+ 181us) server_negotiation.cc:299] Negotiation successful
0806 03:05:39.793290 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":748,"thread_start_us":440,"threads_started":1}
I20260806 03:05:39.795002 22743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28425b83f776439587221a765fcdfa53: Bootstrap starting.
I20260806 03:05:39.799191 22743 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 28425b83f776439587221a765fcdfa53: Neither blocks nor log segments found. Creating new log.
I20260806 03:05:39.799998 32043 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 03:05:39.800558 22743 log.cc:824] T 00000000000000000000000000000000 P 28425b83f776439587221a765fcdfa53: Log is configured to *not* fsync() on all Append() calls
I20260806 03:05:39.804366 22743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28425b83f776439587221a765fcdfa53: No bootstrap required, opened a new log
I20260806 03:05:39.808302 22746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:39.805264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59665 (local address 127.31.74.254:36377)
0806 03:05:39.805525 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:39.805533 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:39.805943 (+ 410us) server_negotiation.cc:407] Connection header received
0806 03:05:39.806185 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:39.806189 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:39.806243 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:39.806402 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:39.807489 (+ 1087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:05:39.807502 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:05:39.807506 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 03:05:39.807527 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:05:39.807575 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 03:05:39.807596 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:05:39.807762 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0806 03:05:39.807945 (+ 183us) server_negotiation.cc:299] Negotiation successful
0806 03:05:39.808018 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260806 03:05:39.813679 22743 raft_consensus.cc:348] T 00000000000000000000000000000000 P 28425b83f776439587221a765fcdfa53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28425b83f776439587221a765fcdfa53" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36377 } }
I20260806 03:05:39.814142 22743 raft_consensus.cc:374] T 00000000000000000000000000000000 P 28425b83f776439587221a765fcdfa53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:05:39.814262 22743 raft_consensus.cc:729] T 00000000000000000000000000000000 P 28425b83f776439587221a765fcdfa53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28425b83f776439587221a765fcdfa53, State: Initialized, Role: FOLLOWER
I20260806 03:05:39.815116 22743 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28425b83f776439587221a765fcdfa53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28425b83f776439587221a765fcdfa53" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36377 } }
I20260806 03:05:39.815446 22743 raft_consensus.cc:388] T 00000000000000000000000000000000 P 28425b83f776439587221a765fcdfa53 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:05:39.815635 22743 raft_consensus.cc:482] T 00000000000000000000000000000000 P 28425b83f776439587221a765fcdfa53 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:05:39.815837 22743 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 28425b83f776439587221a765fcdfa53 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:05:39.818750 22743 raft_consensus.cc:504] T 00000000000000000000000000000000 P 28425b83f776439587221a765fcdfa53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28425b83f776439587221a765fcdfa53" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36377 } }
I20260806 03:05:39.819424 22743 leader_election.cc:302] T 00000000000000000000000000000000 P 28425b83f776439587221a765fcdfa53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 28425b83f776439587221a765fcdfa53; no voters:
I20260806 03:05:39.820117 22743 leader_election.cc:288] T 00000000000000000000000000000000 P 28425b83f776439587221a765fcdfa53 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:05:39.820322 22753 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 28425b83f776439587221a765fcdfa53 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:05:39.821094 22753 raft_consensus.cc:686] T 00000000000000000000000000000000 P 28425b83f776439587221a765fcdfa53 [term 1 LEADER]: Becoming Leader. State: Replica: 28425b83f776439587221a765fcdfa53, State: Running, Role: LEADER
I20260806 03:05:39.821746 22753 consensus_queue.cc:237] T 00000000000000000000000000000000 P 28425b83f776439587221a765fcdfa53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28425b83f776439587221a765fcdfa53" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36377 } }
I20260806 03:05:39.822454 22743 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28425b83f776439587221a765fcdfa53 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:05:39.827073 22755 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28425b83f776439587221a765fcdfa53 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28425b83f776439587221a765fcdfa53. Latest consensus state: current_term: 1 leader_uuid: "28425b83f776439587221a765fcdfa53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28425b83f776439587221a765fcdfa53" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36377 } } }
I20260806 03:05:39.827608 22754 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28425b83f776439587221a765fcdfa53 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "28425b83f776439587221a765fcdfa53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28425b83f776439587221a765fcdfa53" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 36377 } } }
I20260806 03:05:39.827883 22755 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28425b83f776439587221a765fcdfa53 [sys.catalog]: This master's current role is: LEADER
I20260806 03:05:39.827941 22754 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28425b83f776439587221a765fcdfa53 [sys.catalog]: This master's current role is: LEADER
I20260806 03:05:39.830206 22762 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:05:39.837085 22762 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:05:39.849416 22762 catalog_manager.cc:1347] Generated new cluster ID: 4e47b76094924c93840f0ce5d57693f4
I20260806 03:05:39.849591 22762 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:05:39.891782 22762 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:05:39.892324 22762 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:05:39.905916 22762 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 28425b83f776439587221a765fcdfa53: Generated new TSK 0
I20260806 03:05:39.907058 22762 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 03:05:39.915376 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 22673
W20260806 03:05:39.919104 22738 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:41721
W20260806 03:05:39.920218 32043 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 22673: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 03:05:39.920562 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 22673
2026-08-06T03:05:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (2393 ms)
[----------] 18 tests from AuthTokenIssuingTest (43263 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785985105255475-32043-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 03:05:40 dist-test-slave-dhph krb5kdc[22775](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 03:05:40 dist-test-slave-dhph krb5kdc[22775](info): set up 2 sockets
Aug 06 03:05:40 dist-test-slave-dhph krb5kdc[22775](info): commencing operation
krb5kdc: starting...
W20260806 03:05:41.417450 32043 mini_kdc.cc:121] Time spent starting KDC: real 1.446s user 0.002s sys 0.071s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785985105255475-32043-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785985105255475-32043-0/krb5kdc/test-user.keytab.
Aug 06 03:05:41 dist-test-slave-dhph krb5kdc[22775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985541, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T03:05:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:05:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.74.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab.
Entry for principal kudu/127.31.74.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab.
I20260806 03:05:41.894081 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785985105255475-32043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785985105255475-32043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785985105255475-32043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:37453
--webserver_interface=127.31.74.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab
--principal=kudu/127.31.74.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:46485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.74.254:37453,127.31.74.253:37047,127.31.74.252:41949
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:05:42.331254 22789 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 03:05:42.331609 22789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:05:42.331684 22789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:05:42.331773 22789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:05:42.340281 22789 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:05:42.340410 22789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:05:42.340479 22789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:05:42.340530 22789 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:05:42.340603 22789 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:05:42.351975 22789 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:46485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785985105255475-32043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785985105255475-32043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.74.254:37453,127.31.74.253:37047,127.31.74.252:41949
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab
--principal=kudu/127.31.74.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:37453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.74.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785985105255475-32043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:05:42.353631 22789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:05:42.355249 22789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:05:42.369215 22794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:05:42.369544 22795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:05:42.369624 22797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:05:42.371062 22789 server_base.cc:1034] running on GCE node
I20260806 03:05:42.372087 22789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:05:42.373719 22789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:05:42.375061 22789 hybrid_clock.cc:648] HybridClock initialized: now 1785985542374974 us; error 35 us; skew 500 ppm
I20260806 03:05:42.375711 22789 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 03:05:42 dist-test-slave-dhph krb5kdc[22775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985542, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:05:42.381975 22789 init.cc:377] Logged in from keytab as kudu/127.31.74.254@KRBTEST.COM (short username kudu)
I20260806 03:05:42.384899 22789 webserver.cc:460] Webserver started at http://127.31.74.254:41715/ using document root <none> and password file <none>
I20260806 03:05:42.385982 22789 fs_manager.cc:359] Metadata directory not provided
I20260806 03:05:42.386147 22789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:05:42.386535 22789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:05:42.390942 22789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785985105255475-32043-0/minicluster-data/master-0/data/instance:
uuid: "63ab3bf44b344bfd8d2c047862544cd2"
format_stamp: "Formatted at 2026-08-06 03:05:42 on dist-test-slave-dhph"
server_key: "b32a41be151a6e0f6e84cd9bebfc8e28"
server_key_iv: "85036df4458918e76e945b39a0ebd2d2"
server_key_version: "encryptionkey@0"
I20260806 03:05:42.392434 22789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785985105255475-32043-0/minicluster-data/master-0/wal/instance:
uuid: "63ab3bf44b344bfd8d2c047862544cd2"
format_stamp: "Formatted at 2026-08-06 03:05:42 on dist-test-slave-dhph"
server_key: "b32a41be151a6e0f6e84cd9bebfc8e28"
server_key_iv: "85036df4458918e76e945b39a0ebd2d2"
server_key_version: "encryptionkey@0"
I20260806 03:05:42.400421 22789 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.000s sys 0.008s
I20260806 03:05:42.405223 22804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:05:42.407316 22789 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260806 03:05:42.407578 22789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785985105255475-32043-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785985105255475-32043-0/minicluster-data/master-0/wal
uuid: "63ab3bf44b344bfd8d2c047862544cd2"
format_stamp: "Formatted at 2026-08-06 03:05:42 on dist-test-slave-dhph"
server_key: "b32a41be151a6e0f6e84cd9bebfc8e28"
server_key_iv: "85036df4458918e76e945b39a0ebd2d2"
server_key_version: "encryptionkey@0"
I20260806 03:05:42.407876 22789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785985105255475-32043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785985105255475-32043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785985105255475-32043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:05:42.436580 22789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:05:42.442171 22789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:05:42.442525 22789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:05:42.467029 22789 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.254:37453
I20260806 03:05:42.467051 22859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.254:37453 every 8 connection(s)
I20260806 03:05:42.468950 22789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
I20260806 03:05:42.473417 22860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:05:42.474771 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 22789
I20260806 03:05:42.475035 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785985105255475-32043-0/minicluster-data/master-0/wal/instance
I20260806 03:05:42.475992 32043 external_mini_cluster.cc:1428] Setting key 99006b943f30442544aee7b1c1d6a402
I20260806 03:05:42.485893 22860 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.74.254" port: 37453 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 03:05:42 dist-test-slave-dhph krb5kdc[22775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985542, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.254@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.74.253@KRBTEST.COM; defaulting to no policy
I20260806 03:05:42.520229 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:42.490655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:37453 (local address 127.0.0.1:45688)
0806 03:05:42.491716 (+ 1061us) negotiation.cc:107] Waiting for socket to connect
0806 03:05:42.491781 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0806 03:05:42.495333 (+ 3552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:05:42.497130 (+ 1797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:05:42.497137 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:05:42.497826 (+ 689us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:05:42.498242 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:05:42.498264 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:05:42.501229 (+ 2965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:05:42.501234 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:05:42.502507 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:05:42.502528 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:05:42.504336 (+ 1808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:05:42.504343 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:05:42.504501 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:05:42.505426 (+ 925us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:05:42.505461 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:05:42.508749 (+ 3288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:05:42.514836 (+ 6087us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:05:42.514849 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:05:42.514865 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:05:42.515906 (+ 1041us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:05:42.516565 (+ 659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:05:42.516569 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:05:42.516571 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:05:42.516749 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:05:42.517988 (+ 1239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:05:42.517995 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:05:42.518251 (+ 256us) client_negotiation.cc:769] Sending connection context
0806 03:05:42.518502 (+ 251us) client_negotiation.cc:240] Negotiation successful
0806 03:05:42.519039 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":722,"thread_start_us":433,"threads_started":1}
I20260806 03:05:42.520251 22863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:42.490655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45688 (local address 127.31.74.254:37453)
0806 03:05:42.492698 (+ 2043us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:42.492704 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:42.492747 (+ 43us) server_negotiation.cc:407] Connection header received
0806 03:05:42.496110 (+ 3363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:42.496123 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:42.496666 (+ 543us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:42.497195 (+ 529us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:42.498552 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:42.500988 (+ 2436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:42.502835 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:42.504037 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:42.504279 (+ 242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:05:42.509077 (+ 4798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:05:42.509094 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:05:42.509118 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:05:42.509181 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:05:42.514627 (+ 5446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:05:42.516103 (+ 1476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:05:42.516117 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:05:42.516123 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:05:42.516372 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:05:42.516915 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:05:42.516922 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:05:42.516928 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:05:42.517776 (+ 848us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:05:42.518031 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0806 03:05:42.518754 (+ 723us) server_negotiation.cc:299] Negotiation successful
0806 03:05:42.519272 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":154,"server-negotiator.queue_time_us":1833,"spinlock_wait_cycles":61312,"thread_start_us":1584,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.253@KRBTEST.COM" created.
I20260806 03:05:42.523729 22860 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.74.253" port: 37047 } has no permanent_uuid. Determining permanent_uuid...
I20260806 03:05:42.525629 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:42.525141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.253:37047 (local address 127.0.0.1:35774)
0806 03:05:42.525372 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0806 03:05:42.525497 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.253:37047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
W20260806 03:05:42.525988 22809 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.74.253:37047: connect: Connection refused (error 111)
I20260806 03:05:42.528038 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:42.527543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.253:37047 (local address 127.0.0.1:35790)
0806 03:05:42.527780 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0806 03:05:42.527882 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.253:37047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260806 03:05:42.528293 22860 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.74.253:37047: Network error: Client connection negotiation failed: client connection to 127.31.74.253:37047: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.253.keytab.
Entry for principal kudu/127.31.74.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.253.keytab.
I20260806 03:05:42.572335 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785985105255475-32043-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785985105255475-32043-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785985105255475-32043-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785985105255475-32043-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.253:37047
--webserver_interface=127.31.74.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.253.keytab
--principal=kudu/127.31.74.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:46485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.74.254:37453,127.31.74.253:37047,127.31.74.252:41949
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 03:05:42.577688 22860 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.74.253" port: 37047 } attempt: 1
I20260806 03:05:42.579134 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:42.578447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.253:37047 (local address 127.0.0.1:35804)
0806 03:05:42.578763 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0806 03:05:42.578926 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.253:37047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260806 03:05:42.580544 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:42.580083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.253:37047 (local address 127.0.0.1:35818)
0806 03:05:42.580268 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0806 03:05:42.580361 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.253:37047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260806 03:05:42.580822 22860 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.74.253:37047: Network error: Client connection negotiation failed: client connection to 127.31.74.253:37047: connect: Connection refused (error 111)
I20260806 03:05:42.649142 22860 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.74.253" port: 37047 } attempt: 2
I20260806 03:05:42.650568 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:42.649984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.253:37047 (local address 127.0.0.1:35824)
0806 03:05:42.650309 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0806 03:05:42.650406 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.253:37047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260806 03:05:42.651973 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:42.651544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.253:37047 (local address 127.0.0.1:35826)
0806 03:05:42.651706 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0806 03:05:42.651805 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.253:37047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260806 03:05:42.652294 22860 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.74.253:37047: Network error: Client connection negotiation failed: client connection to 127.31.74.253:37047: connect: Connection refused (error 111)
I20260806 03:05:42.743775 22860 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.74.253" port: 37047 } attempt: 3
I20260806 03:05:42.745102 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:42.744586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.253:37047 (local address 127.0.0.1:35834)
0806 03:05:42.744859 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0806 03:05:42.744958 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.253:37047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
I20260806 03:05:42.746357 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:42.745922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.253:37047 (local address 127.0.0.1:35844)
0806 03:05:42.746091 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0806 03:05:42.746188 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.253:37047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260806 03:05:42.746654 22860 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.74.253:37047: Network error: Client connection negotiation failed: client connection to 127.31.74.253:37047: connect: Connection refused (error 111)
I20260806 03:05:42.889976 22860 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.74.253" port: 37047 } attempt: 4
I20260806 03:05:42.891821 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:42.891029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.253:37047 (local address 127.0.0.1:35846)
0806 03:05:42.891482 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0806 03:05:42.891620 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.253:37047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256}
I20260806 03:05:42.893777 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:42.893136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.253:37047 (local address 127.0.0.1:35862)
0806 03:05:42.893411 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0806 03:05:42.893529 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.253:37047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
W20260806 03:05:42.894115 22860 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.74.253:37047: Network error: Client connection negotiation failed: client connection to 127.31.74.253:37047: connect: Connection refused (error 111)
W20260806 03:05:42.999411 22867 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 03:05:42.999801 22867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:05:42.999938 22867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:05:43.000061 22867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:05:43.009743 22867 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:05:43.009888 22867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:05:43.009994 22867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:05:43.010088 22867 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:05:43.010213 22867 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:05:43.022130 22867 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:46485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785985105255475-32043-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785985105255475-32043-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.31.74.254:37453,127.31.74.253:37047,127.31.74.252:41949
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.253.keytab
--principal=kudu/127.31.74.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.253:37047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785985105255475-32043-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.31.74.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785985105255475-32043-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:05:43.023859 22867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:05:43.025393 22867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:05:43.042361 22872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:05:43.042438 22875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:05:43.042835 22873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:05:43.043601 22867 server_base.cc:1034] running on GCE node
I20260806 03:05:43.044615 22867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:05:43.046319 22867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:05:43.047645 22867 hybrid_clock.cc:648] HybridClock initialized: now 1785985543047523 us; error 98 us; skew 500 ppm
I20260806 03:05:43.048090 22867 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 03:05:43 dist-test-slave-dhph krb5kdc[22775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985543, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:05:43.053136 22867 init.cc:377] Logged in from keytab as kudu/127.31.74.253@KRBTEST.COM (short username kudu)
I20260806 03:05:43.055218 22867 webserver.cc:460] Webserver started at http://127.31.74.253:34927/ using document root <none> and password file <none>
I20260806 03:05:43.056316 22867 fs_manager.cc:359] Metadata directory not provided
I20260806 03:05:43.056470 22867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:05:43.056830 22867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:05:43.060237 22867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785985105255475-32043-0/minicluster-data/master-1/data/instance:
uuid: "74a766d5df9a455f839d7667a558efeb"
format_stamp: "Formatted at 2026-08-06 03:05:43 on dist-test-slave-dhph"
server_key: "6fe00321f7f000c5e95043d34ed074e8"
server_key_iv: "6fa5eee28ed4cb5d08c77c0d9b37ef0d"
server_key_version: "encryptionkey@0"
I20260806 03:05:43.061134 22867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785985105255475-32043-0/minicluster-data/master-1/wal/instance:
uuid: "74a766d5df9a455f839d7667a558efeb"
format_stamp: "Formatted at 2026-08-06 03:05:43 on dist-test-slave-dhph"
server_key: "6fe00321f7f000c5e95043d34ed074e8"
server_key_iv: "6fa5eee28ed4cb5d08c77c0d9b37ef0d"
server_key_version: "encryptionkey@0"
I20260806 03:05:43.067524 22867 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260806 03:05:43.072273 22882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:05:43.073961 22867 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260806 03:05:43.074143 22867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785985105255475-32043-0/minicluster-data/master-1/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785985105255475-32043-0/minicluster-data/master-1/wal
uuid: "74a766d5df9a455f839d7667a558efeb"
format_stamp: "Formatted at 2026-08-06 03:05:43 on dist-test-slave-dhph"
server_key: "6fe00321f7f000c5e95043d34ed074e8"
server_key_iv: "6fa5eee28ed4cb5d08c77c0d9b37ef0d"
server_key_version: "encryptionkey@0"
I20260806 03:05:43.074375 22867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785985105255475-32043-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785985105255475-32043-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785985105255475-32043-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:05:43.089288 22867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:05:43.095873 22867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:05:43.096283 22867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:05:43.124001 22936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.253:37047 every 8 connection(s)
I20260806 03:05:43.124078 22867 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.253:37047
I20260806 03:05:43.128101 22867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785985105255475-32043-0/minicluster-data/master-1/data/info.pb
I20260806 03:05:43.133024 22937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:05:43.138078 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 22867
I20260806 03:05:43.138312 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785985105255475-32043-0/minicluster-data/master-1/wal/instance
I20260806 03:05:43.138948 32043 external_mini_cluster.cc:1428] Setting key 45ca290bddda2aefc37a69f964fa5ec2
I20260806 03:05:43.142210 22937 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.74.254" port: 37453 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 03:05:43 dist-test-slave-dhph krb5kdc[22775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985543, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.253@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 03:05:43.175446 22939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:43.146267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45698 (local address 127.31.74.254:37453)
0806 03:05:43.146945 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:43.146953 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:43.148158 (+ 1205us) server_negotiation.cc:407] Connection header received
0806 03:05:43.151923 (+ 3765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:43.151929 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:43.152042 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:43.152164 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:43.154929 (+ 2765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:43.155765 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:43.157780 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:43.159239 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:43.159666 (+ 427us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:05:43.165498 (+ 5832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:05:43.165504 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:05:43.165508 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:05:43.165561 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:05:43.170695 (+ 5134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:05:43.172291 (+ 1596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:05:43.172297 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:05:43.172300 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:05:43.172473 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:05:43.173100 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:05:43.173104 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:05:43.173106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:05:43.173440 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:05:43.173582 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0806 03:05:43.174892 (+ 1310us) server_negotiation.cc:299] Negotiation successful
0806 03:05:43.175124 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":361,"threads_started":1}
I20260806 03:05:43.177129 22940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:43.147004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:37453 (local address 127.0.0.1:45698)
0806 03:05:43.147919 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0806 03:05:43.147985 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0806 03:05:43.151022 (+ 3037us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:05:43.152506 (+ 1484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:05:43.152523 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:05:43.153502 (+ 979us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:05:43.154453 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:05:43.154485 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:05:43.155958 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:05:43.155967 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:05:43.157466 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:05:43.157481 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:05:43.159806 (+ 2325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:05:43.159813 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:05:43.160219 (+ 406us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:05:43.161672 (+ 1453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:05:43.161700 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:05:43.165081 (+ 3381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:05:43.171053 (+ 5972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:05:43.171070 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:05:43.171097 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:05:43.172049 (+ 952us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:05:43.172642 (+ 593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:05:43.172651 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:05:43.172658 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:05:43.172939 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:05:43.173626 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:05:43.173639 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:05:43.174205 (+ 566us) client_negotiation.cc:769] Sending connection context
0806 03:05:43.174645 (+ 440us) client_negotiation.cc:240] Negotiation successful
0806 03:05:43.175441 (+ 796us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":353,"threads_started":1}
I20260806 03:05:43.179751 22937 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.74.253" port: 37047 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.31.74.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.252@KRBTEST.COM" created.
Aug 06 03:05:43 dist-test-slave-dhph krb5kdc[22775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985543, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.253@KRBTEST.COM for kudu/127.31.74.253@KRBTEST.COM
I20260806 03:05:43.193428 22860 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.74.253" port: 37047 } attempt: 5
I20260806 03:05:43.203780 22940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:43.181155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.253:37047 (local address 127.0.0.1:35864)
0806 03:05:43.181422 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0806 03:05:43.181452 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0806 03:05:43.181793 (+ 341us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:05:43.183007 (+ 1214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:05:43.183012 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:05:43.184175 (+ 1163us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:05:43.184570 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:05:43.184596 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:05:43.187386 (+ 2790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:05:43.187392 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:05:43.188305 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:05:43.188321 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:05:43.189937 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:05:43.189947 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:05:43.190123 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:05:43.190911 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:05:43.190931 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:05:43.193107 (+ 2176us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:05:43.199102 (+ 5995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:05:43.199107 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:05:43.199111 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:05:43.199678 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:05:43.200324 (+ 646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:05:43.200330 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:05:43.200335 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:05:43.200859 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:05:43.201990 (+ 1131us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:05:43.201998 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:05:43.202204 (+ 206us) client_negotiation.cc:769] Sending connection context
0806 03:05:43.202403 (+ 199us) client_negotiation.cc:240] Negotiation successful
0806 03:05:43.202843 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79,"mutex_wait_us":4}
I20260806 03:05:43.203868 22942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:43.181331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35864 (local address 127.31.74.253:37047)
0806 03:05:43.182412 (+ 1081us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:43.182420 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:43.182449 (+ 29us) server_negotiation.cc:407] Connection header received
0806 03:05:43.182513 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:43.182519 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:43.182824 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:43.183021 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:43.185012 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:43.187104 (+ 2092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:43.188593 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:43.189722 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:43.189910 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:05:43.193394 (+ 3484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:05:43.193415 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:05:43.193440 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:05:43.193485 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:05:43.198934 (+ 5449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:05:43.199844 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:05:43.199857 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:05:43.199863 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:05:43.200027 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:05:43.201106 (+ 1079us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:05:43.201113 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:05:43.201119 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:05:43.201754 (+ 635us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:05:43.202012 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0806 03:05:43.202829 (+ 817us) server_negotiation.cc:299] Negotiation successful
0806 03:05:43.203240 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":864,"thread_start_us":664,"threads_started":1}
Aug 06 03:05:43 dist-test-slave-dhph krb5kdc[22775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985542, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.254@KRBTEST.COM for kudu/127.31.74.253@KRBTEST.COM
I20260806 03:05:43.207023 22937 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.74.252" port: 41949 } has no permanent_uuid. Determining permanent_uuid...
I20260806 03:05:43.208585 22940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:43.207953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.252:41949 (local address 127.0.0.1:57946)
0806 03:05:43.208155 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0806 03:05:43.208340 (+ 185us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.252:41949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260806 03:05:43.208977 22884 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.74.252:41949: connect: Connection refused (error 111)
I20260806 03:05:43.211043 22940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:43.210518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.252:41949 (local address 127.0.0.1:57962)
0806 03:05:43.210759 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0806 03:05:43.210875 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.252:41949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260806 03:05:43.211354 22937 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.74.252:41949: Network error: Client connection negotiation failed: client connection to 127.31.74.252:41949: connect: Connection refused (error 111)
I20260806 03:05:43.216012 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:43.194215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.253:37047 (local address 127.0.0.1:35866)
0806 03:05:43.194573 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0806 03:05:43.194624 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 03:05:43.194763 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:05:43.195502 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:05:43.195507 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:05:43.196055 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:05:43.196353 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:05:43.196367 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:05:43.199023 (+ 2656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:05:43.199029 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:05:43.200898 (+ 1869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:05:43.200931 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:05:43.202464 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:05:43.202474 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:05:43.202864 (+ 390us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:05:43.203615 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:05:43.203644 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:05:43.206965 (+ 3321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:05:43.211361 (+ 4396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:05:43.211367 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:05:43.211372 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:05:43.211871 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:05:43.212285 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:05:43.212291 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:05:43.212294 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:05:43.212443 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:05:43.213025 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:05:43.213030 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:05:43.213546 (+ 516us) client_negotiation.cc:769] Sending connection context
0806 03:05:43.214367 (+ 821us) client_negotiation.cc:240] Negotiation successful
0806 03:05:43.215453 (+ 1086us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207}
I20260806 03:05:43.216012 22943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:43.194368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35866 (local address 127.31.74.253:37047)
0806 03:05:43.195120 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:43.195125 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:43.195175 (+ 50us) server_negotiation.cc:407] Connection header received
0806 03:05:43.195242 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:43.195248 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:43.195338 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:43.195478 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:43.196568 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:43.198850 (+ 2282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:43.201290 (+ 2440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:43.202206 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:43.202393 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:05:43.207265 (+ 4872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:05:43.207272 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:05:43.207277 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:05:43.207318 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:05:43.211173 (+ 3855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:05:43.212000 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:05:43.212007 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:05:43.212011 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:05:43.212158 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:05:43.212567 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:05:43.212572 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:05:43.212575 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:05:43.212880 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:05:43.213045 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0806 03:05:43.214477 (+ 1432us) server_negotiation.cc:299] Negotiation successful
0806 03:05:43.215385 (+ 908us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":362,"threads_started":1}
I20260806 03:05:43.218192 22860 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.74.252" port: 41949 } has no permanent_uuid. Determining permanent_uuid...
I20260806 03:05:43.219846 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:43.219270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.252:41949 (local address 127.0.0.1:57974)
0806 03:05:43.219492 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0806 03:05:43.219675 (+ 183us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.252:41949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260806 03:05:43.220880 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:43.220537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.252:41949 (local address 127.0.0.1:57986)
0806 03:05:43.220685 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0806 03:05:43.220763 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.252:41949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260806 03:05:43.221366 22860 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.74.252:41949: Network error: Client connection negotiation failed: client connection to 127.31.74.252:41949: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.252.keytab.
Entry for principal kudu/127.31.74.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.252.keytab.
W20260806 03:05:43.240933 32043 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.31.74.252: real 0.101s user 0.003s sys 0.033s
I20260806 03:05:43.241806 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785985105255475-32043-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785985105255475-32043-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785985105255475-32043-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785985105255475-32043-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.252:41949
--webserver_interface=127.31.74.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.252.keytab
--principal=kudu/127.31.74.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:46485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.74.254:37453,127.31.74.253:37047,127.31.74.252:41949
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 03:05:43.260777 22937 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.74.252" port: 41949 } attempt: 1
I20260806 03:05:43.262225 22940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:43.261619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.252:41949 (local address 127.0.0.1:58000)
0806 03:05:43.261916 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0806 03:05:43.262044 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.252:41949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160}
I20260806 03:05:43.263427 22940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:43.263026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.252:41949 (local address 127.0.0.1:58002)
0806 03:05:43.263225 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0806 03:05:43.263308 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.252:41949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260806 03:05:43.263721 22937 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.74.252:41949: Network error: Client connection negotiation failed: client connection to 127.31.74.252:41949: connect: Connection refused (error 111)
I20260806 03:05:43.272797 22860 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.74.252" port: 41949 } attempt: 1
I20260806 03:05:43.274915 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:43.273934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.252:41949 (local address 127.0.0.1:58010)
0806 03:05:43.274498 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0806 03:05:43.274668 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.252:41949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219}
I20260806 03:05:43.277350 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:43.276645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.252:41949 (local address 127.0.0.1:58012)
0806 03:05:43.277023 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0806 03:05:43.277144 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.252:41949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
W20260806 03:05:43.277813 22860 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.74.252:41949: Network error: Client connection negotiation failed: client connection to 127.31.74.252:41949: connect: Connection refused (error 111)
I20260806 03:05:43.332051 22937 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.74.252" port: 41949 } attempt: 2
I20260806 03:05:43.333985 22940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:43.333021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.252:41949 (local address 127.0.0.1:58014)
0806 03:05:43.333323 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0806 03:05:43.333688 (+ 365us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.252:41949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260806 03:05:43.336037 22940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:43.335268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.252:41949 (local address 127.0.0.1:58018)
0806 03:05:43.335445 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0806 03:05:43.335547 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.252:41949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260806 03:05:43.336486 22937 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.74.252:41949: Network error: Client connection negotiation failed: client connection to 127.31.74.252:41949: connect: Connection refused (error 111)
I20260806 03:05:43.346203 22860 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.74.252" port: 41949 } attempt: 2
I20260806 03:05:43.348999 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:43.347736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.252:41949 (local address 127.0.0.1:58020)
0806 03:05:43.348255 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0806 03:05:43.348482 (+ 227us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.252:41949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260806 03:05:43.351333 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:43.350460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.252:41949 (local address 127.0.0.1:58026)
0806 03:05:43.350896 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0806 03:05:43.351018 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.252:41949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210}
W20260806 03:05:43.351913 22860 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.74.252:41949: Network error: Client connection negotiation failed: client connection to 127.31.74.252:41949: connect: Connection refused (error 111)
I20260806 03:05:43.427882 22937 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.74.252" port: 41949 } attempt: 3
I20260806 03:05:43.429939 22940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:43.429179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.252:41949 (local address 127.0.0.1:58040)
0806 03:05:43.429440 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0806 03:05:43.429727 (+ 287us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.252:41949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260806 03:05:43.431361 22940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:43.430849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.252:41949 (local address 127.0.0.1:58044)
0806 03:05:43.431124 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0806 03:05:43.431220 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.252:41949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139}
W20260806 03:05:43.431694 22937 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.74.252:41949: Network error: Client connection negotiation failed: client connection to 127.31.74.252:41949: connect: Connection refused (error 111)
I20260806 03:05:43.458240 22860 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.74.252" port: 41949 } attempt: 3
I20260806 03:05:43.459830 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:43.459219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.252:41949 (local address 127.0.0.1:58054)
0806 03:05:43.459474 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0806 03:05:43.459582 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.252:41949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260806 03:05:43.461382 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:43.460936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.252:41949 (local address 127.0.0.1:58066)
0806 03:05:43.461179 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0806 03:05:43.461267 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.252:41949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
W20260806 03:05:43.461681 22860 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.74.252:41949: Network error: Client connection negotiation failed: client connection to 127.31.74.252:41949: connect: Connection refused (error 111)
I20260806 03:05:43.575043 22937 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.74.252" port: 41949 } attempt: 4
I20260806 03:05:43.577158 22940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:43.576396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.252:41949 (local address 127.0.0.1:58082)
0806 03:05:43.576740 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0806 03:05:43.576901 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.252:41949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
I20260806 03:05:43.578994 22940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:43.578517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.252:41949 (local address 127.0.0.1:58094)
0806 03:05:43.578740 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0806 03:05:43.578841 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.252:41949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260806 03:05:43.579435 22937 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.74.252:41949: Network error: Client connection negotiation failed: client connection to 127.31.74.252:41949: connect: Connection refused (error 111)
I20260806 03:05:43.638967 22860 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.74.252" port: 41949 } attempt: 4
I20260806 03:05:43.640409 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:43.639924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.252:41949 (local address 127.0.0.1:58104)
0806 03:05:43.640128 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0806 03:05:43.640227 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.252:41949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260806 03:05:43.641566 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:43.641169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.252:41949 (local address 127.0.0.1:58106)
0806 03:05:43.641372 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0806 03:05:43.641445 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.252:41949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260806 03:05:43.641832 22860 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.74.252:41949: Network error: Client connection negotiation failed: client connection to 127.31.74.252:41949: connect: Connection refused (error 111)
W20260806 03:05:43.672783 22946 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 03:05:43.673192 22946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:05:43.673332 22946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:05:43.673429 22946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:05:43.681627 22946 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:05:43.681761 22946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:05:43.681849 22946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:05:43.681931 22946 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:05:43.682343 22946 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:05:43.694047 22946 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:46485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785985105255475-32043-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785985105255475-32043-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.31.74.254:37453,127.31.74.253:37047,127.31.74.252:41949
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.252.keytab
--principal=kudu/127.31.74.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.252:41949
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785985105255475-32043-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.31.74.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785985105255475-32043-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:05:43.695581 22946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:05:43.697093 22946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:05:43.710916 22951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:05:43.710851 22952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:05:43.710812 22955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:05:43.713001 22946 server_base.cc:1034] running on GCE node
I20260806 03:05:43.714458 22946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:05:43.716184 22946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:05:43.717514 22946 hybrid_clock.cc:648] HybridClock initialized: now 1785985543717432 us; error 52 us; skew 500 ppm
I20260806 03:05:43.718150 22946 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 03:05:43 dist-test-slave-dhph krb5kdc[22775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985543, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:05:43.722110 22946 init.cc:377] Logged in from keytab as kudu/127.31.74.252@KRBTEST.COM (short username kudu)
I20260806 03:05:43.723852 22946 webserver.cc:460] Webserver started at http://127.31.74.252:38649/ using document root <none> and password file <none>
I20260806 03:05:43.724560 22946 fs_manager.cc:359] Metadata directory not provided
I20260806 03:05:43.724650 22946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:05:43.724931 22946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:05:43.728832 22946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785985105255475-32043-0/minicluster-data/master-2/data/instance:
uuid: "a2270660ab23436c9d79e552d4e188bf"
format_stamp: "Formatted at 2026-08-06 03:05:43 on dist-test-slave-dhph"
server_key: "f0c96d1687dc45900f6f934a671cd5b2"
server_key_iv: "f9b4c9020345d22a5ebb5a339120cf88"
server_key_version: "encryptionkey@0"
I20260806 03:05:43.729699 22946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785985105255475-32043-0/minicluster-data/master-2/wal/instance:
uuid: "a2270660ab23436c9d79e552d4e188bf"
format_stamp: "Formatted at 2026-08-06 03:05:43 on dist-test-slave-dhph"
server_key: "f0c96d1687dc45900f6f934a671cd5b2"
server_key_iv: "f9b4c9020345d22a5ebb5a339120cf88"
server_key_version: "encryptionkey@0"
I20260806 03:05:43.737258 22946 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260806 03:05:43.743158 22962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:05:43.745424 22946 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260806 03:05:43.745704 22946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785985105255475-32043-0/minicluster-data/master-2/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785985105255475-32043-0/minicluster-data/master-2/wal
uuid: "a2270660ab23436c9d79e552d4e188bf"
format_stamp: "Formatted at 2026-08-06 03:05:43 on dist-test-slave-dhph"
server_key: "f0c96d1687dc45900f6f934a671cd5b2"
server_key_iv: "f9b4c9020345d22a5ebb5a339120cf88"
server_key_version: "encryptionkey@0"
I20260806 03:05:43.746014 22946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785985105255475-32043-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785985105255475-32043-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785985105255475-32043-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:05:43.774384 22946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:05:43.780645 22946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:05:43.780946 22946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:05:43.802977 22946 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.252:41949
I20260806 03:05:43.803017 23013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.252:41949 every 8 connection(s)
I20260806 03:05:43.804494 22946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785985105255475-32043-0/minicluster-data/master-2/data/info.pb
I20260806 03:05:43.808535 23014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:05:43.809324 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 22946
I20260806 03:05:43.809526 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785985105255475-32043-0/minicluster-data/master-2/wal/instance
I20260806 03:05:43.810030 32043 external_mini_cluster.cc:1428] Setting key dae3473cadf66fba2545b9604d36ff98
I20260806 03:05:43.820449 23014 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.74.254" port: 37453 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 03:05:43 dist-test-slave-dhph krb5kdc[22775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985541, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.74.252@KRBTEST.COM
Aug 06 03:05:43 dist-test-slave-dhph krb5kdc[22775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985543, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.252@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 03:05:43.841127 23017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:43.812954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58116 (local address 127.31.74.252:41949)
0806 03:05:43.814680 (+ 1726us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:43.814719 (+ 39us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:43.814808 (+ 89us) server_negotiation.cc:407] Connection header received
0806 03:05:43.816961 (+ 2153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:43.817029 (+ 68us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:43.817665 (+ 636us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:43.818193 (+ 528us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:43.819954 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:43.822018 (+ 2064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:43.823980 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:43.825113 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:43.825303 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:05:43.831089 (+ 5786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:05:43.831101 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:05:43.831119 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:05:43.831170 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:05:43.835009 (+ 3839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:05:43.836058 (+ 1049us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:05:43.836069 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:05:43.836082 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:05:43.836307 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:05:43.837266 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:05:43.837272 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:05:43.837276 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:05:43.838101 (+ 825us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:05:43.838504 (+ 403us) server_negotiation.cc:1035] Waiting for connection context
0806 03:05:43.839380 (+ 876us) server_negotiation.cc:299] Negotiation successful
0806 03:05:43.839884 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1058,"thread_start_us":562,"threads_started":1}
I20260806 03:05:43.843850 23018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:43.823664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45712 (local address 127.31.74.254:37453)
0806 03:05:43.824294 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:43.824300 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:43.824818 (+ 518us) server_negotiation.cc:407] Connection header received
0806 03:05:43.825067 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:43.825072 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:43.825188 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:43.825360 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:43.826774 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:43.827562 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:43.829083 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:43.829869 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:43.830325 (+ 456us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:05:43.835640 (+ 5315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:05:43.835647 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:05:43.835651 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:05:43.835695 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:05:43.839658 (+ 3963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:05:43.840765 (+ 1107us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:05:43.840771 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:05:43.840774 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:05:43.840934 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:05:43.841782 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:05:43.841787 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:05:43.841790 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:05:43.842177 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:05:43.842369 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0806 03:05:43.843112 (+ 743us) server_negotiation.cc:299] Negotiation successful
0806 03:05:43.843386 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":290,"threads_started":1}
I20260806 03:05:43.843887 23019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:43.823663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:37453 (local address 127.0.0.1:45712)
0806 03:05:43.824646 (+ 983us) negotiation.cc:107] Waiting for socket to connect
0806 03:05:43.824698 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 03:05:43.824880 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:05:43.825436 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:05:43.825444 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:05:43.826232 (+ 788us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:05:43.826530 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:05:43.826546 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:05:43.827732 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:05:43.827738 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:05:43.828901 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:05:43.828915 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:05:43.830674 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:05:43.830724 (+ 50us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:05:43.831030 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:05:43.831975 (+ 945us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:05:43.831994 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:05:43.835336 (+ 3342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:05:43.840041 (+ 4705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:05:43.840063 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:05:43.840069 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:05:43.840617 (+ 548us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:05:43.841204 (+ 587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:05:43.841210 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:05:43.841213 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:05:43.841605 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:05:43.842386 (+ 781us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:05:43.842393 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:05:43.842684 (+ 291us) client_negotiation.cc:769] Sending connection context
0806 03:05:43.842979 (+ 295us) client_negotiation.cc:240] Negotiation successful
0806 03:05:43.843406 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":717,"thread_start_us":541,"threads_started":1}
I20260806 03:05:43.845849 23014 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.74.253" port: 37047 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 03:05:43 dist-test-slave-dhph krb5kdc[22775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985541, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
Aug 06 03:05:43 dist-test-slave-dhph krb5kdc[22775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985543, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.252@KRBTEST.COM for kudu/127.31.74.253@KRBTEST.COM
I20260806 03:05:43.865756 23019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:43.847075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.253:37047 (local address 127.0.0.1:35868)
0806 03:05:43.847615 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0806 03:05:43.847640 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 03:05:43.847778 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:05:43.848163 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:05:43.848170 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:05:43.848756 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:05:43.848997 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:05:43.849020 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:05:43.850094 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:05:43.850102 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:05:43.851555 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:05:43.851574 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:05:43.852836 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:05:43.852849 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:05:43.853035 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:05:43.853707 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:05:43.853727 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:05:43.857277 (+ 3550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:05:43.862021 (+ 4744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:05:43.862027 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:05:43.862030 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:05:43.863080 (+ 1050us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:05:43.863591 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:05:43.863599 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:05:43.863606 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:05:43.863812 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:05:43.864562 (+ 750us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:05:43.864571 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:05:43.864799 (+ 228us) client_negotiation.cc:769] Sending connection context
0806 03:05:43.864975 (+ 176us) client_negotiation.cc:240] Negotiation successful
0806 03:05:43.865385 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383}
I20260806 03:05:43.865916 23020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:43.847199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35868 (local address 127.31.74.253:37047)
0806 03:05:43.847715 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:43.847720 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:43.847738 (+ 18us) server_negotiation.cc:407] Connection header received
0806 03:05:43.847915 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:43.847918 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:43.847986 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:43.848086 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:43.849177 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:43.849939 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:43.851773 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:43.852572 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:43.852749 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:05:43.857583 (+ 4834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:05:43.857594 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:05:43.857603 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:05:43.857660 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:05:43.861749 (+ 4089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:05:43.863281 (+ 1532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:05:43.863286 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:05:43.863289 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:05:43.863450 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:05:43.863991 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:05:43.864000 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:05:43.864005 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:05:43.864397 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:05:43.864532 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0806 03:05:43.865144 (+ 612us) server_negotiation.cc:299] Negotiation successful
0806 03:05:43.865360 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":292,"threads_started":1}
I20260806 03:05:43.867173 23014 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.74.252" port: 41949 } has no permanent_uuid. Determining permanent_uuid...
I20260806 03:05:43.870940 23018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:43.844682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45716 (local address 127.31.74.254:37453)
0806 03:05:43.844866 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:43.844875 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:43.844895 (+ 20us) server_negotiation.cc:407] Connection header received
0806 03:05:43.845090 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:43.845095 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:43.845195 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:43.845325 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:43.847290 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:43.848770 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:43.850186 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:43.850761 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:43.850909 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:05:43.862932 (+ 12023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:05:43.862950 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:05:43.862961 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:05:43.863021 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:05:43.867506 (+ 4485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:05:43.868558 (+ 1052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:05:43.868566 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:05:43.868570 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:05:43.868733 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:05:43.869191 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:05:43.869197 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:05:43.869201 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:05:43.869551 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:05:43.869934 (+ 383us) server_negotiation.cc:1035] Waiting for connection context
0806 03:05:43.870291 (+ 357us) server_negotiation.cc:299] Negotiation successful
0806 03:05:43.870559 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
Aug 06 03:05:43 dist-test-slave-dhph krb5kdc[22775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985543, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.252@KRBTEST.COM for kudu/127.31.74.252@KRBTEST.COM
I20260806 03:05:43.878684 22937 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.74.252" port: 41949 } attempt: 5
Aug 06 03:05:43 dist-test-slave-dhph krb5kdc[22775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985541, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.74.253@KRBTEST.COM
I20260806 03:05:43.888191 23019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:43.868771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.252:41949 (local address 127.0.0.1:58120)
0806 03:05:43.869061 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0806 03:05:43.869077 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 03:05:43.869235 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:05:43.869555 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:05:43.869559 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:05:43.870051 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:05:43.870222 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:05:43.870232 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:05:43.871751 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:05:43.871760 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:05:43.873593 (+ 1833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:05:43.873608 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:05:43.874397 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:05:43.874403 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:05:43.874672 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:05:43.876145 (+ 1473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:05:43.876170 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:05:43.879441 (+ 3271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:05:43.884139 (+ 4698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:05:43.884146 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:05:43.884150 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:05:43.884986 (+ 836us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:05:43.885921 (+ 935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:05:43.885928 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:05:43.885934 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:05:43.886094 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:05:43.887009 (+ 915us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:05:43.887017 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:05:43.887193 (+ 176us) client_negotiation.cc:769] Sending connection context
0806 03:05:43.887377 (+ 184us) client_negotiation.cc:240] Negotiation successful
0806 03:05:43.887743 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177}
I20260806 03:05:43.888469 23017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:43.868778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58120 (local address 127.31.74.252:41949)
0806 03:05:43.868997 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:43.869004 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:43.869153 (+ 149us) server_negotiation.cc:407] Connection header received
0806 03:05:43.869354 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:43.869360 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:43.869428 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:43.869538 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:43.870376 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:43.871573 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:43.873777 (+ 2204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:43.874203 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:43.874370 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:05:43.879681 (+ 5311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:05:43.879690 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:05:43.879698 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:05:43.879744 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:05:43.883958 (+ 4214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:05:43.885592 (+ 1634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:05:43.885597 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:05:43.885599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:05:43.885798 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:05:43.886251 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:05:43.886260 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:05:43.886266 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:05:43.886851 (+ 585us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:05:43.887032 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0806 03:05:43.887509 (+ 477us) server_negotiation.cc:299] Negotiation successful
0806 03:05:43.887967 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":142,"server-negotiator.queue_time_us":89}
Aug 06 03:05:43 dist-test-slave-dhph krb5kdc[22775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985543, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.253@KRBTEST.COM for kudu/127.31.74.252@KRBTEST.COM
I20260806 03:05:43.896929 23020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:43.875038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35884 (local address 127.31.74.253:37047)
0806 03:05:43.875244 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:43.875252 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:43.875435 (+ 183us) server_negotiation.cc:407] Connection header received
0806 03:05:43.875658 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:43.875664 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:43.875742 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:43.875854 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:43.877539 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:43.878857 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:43.880840 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:43.881576 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:43.881719 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:05:43.887836 (+ 6117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:05:43.887843 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:05:43.887848 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:05:43.887892 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:05:43.892615 (+ 4723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:05:43.893946 (+ 1331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:05:43.893955 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:05:43.893960 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:05:43.894122 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:05:43.894674 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:05:43.894681 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:05:43.894684 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:05:43.895122 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:05:43.895390 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0806 03:05:43.896050 (+ 660us) server_negotiation.cc:299] Negotiation successful
0806 03:05:43.896535 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 03:05:43.897869 23014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2270660ab23436c9d79e552d4e188bf: Bootstrap starting.
I20260806 03:05:43.898993 22940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:43.879386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.252:41949 (local address 127.0.0.1:58136)
0806 03:05:43.879678 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0806 03:05:43.879711 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0806 03:05:43.879824 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:05:43.880961 (+ 1137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:05:43.880968 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:05:43.881494 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:05:43.881660 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:05:43.881669 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:05:43.883235 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:05:43.883241 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:05:43.884376 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:05:43.884393 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:05:43.885725 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:05:43.885732 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:05:43.886023 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:05:43.886751 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:05:43.886779 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:05:43.889981 (+ 3202us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:05:43.895455 (+ 5474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:05:43.895462 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:05:43.895467 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:05:43.896018 (+ 551us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:05:43.896937 (+ 919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:05:43.896942 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:05:43.896945 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:05:43.897110 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:05:43.897841 (+ 731us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:05:43.897849 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:05:43.898027 (+ 178us) client_negotiation.cc:769] Sending connection context
0806 03:05:43.898238 (+ 211us) client_negotiation.cc:240] Negotiation successful
0806 03:05:43.898558 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167}
I20260806 03:05:43.899051 23021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:43.879505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58136 (local address 127.31.74.252:41949)
0806 03:05:43.880577 (+ 1072us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:43.880584 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:43.880609 (+ 25us) server_negotiation.cc:407] Connection header received
0806 03:05:43.880661 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:43.880665 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:43.880760 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:43.880931 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:43.881858 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:43.883075 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:43.884551 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:43.885466 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:43.885676 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:05:43.890290 (+ 4614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:05:43.890296 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:05:43.890300 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:05:43.890364 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:05:43.895275 (+ 4911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:05:43.896190 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:05:43.896196 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:05:43.896200 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:05:43.896753 (+ 553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:05:43.897275 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:05:43.897280 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:05:43.897283 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:05:43.897700 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:05:43.897868 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0806 03:05:43.898354 (+ 486us) server_negotiation.cc:299] Negotiation successful
0806 03:05:43.898609 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":917,"thread_start_us":577,"threads_started":1}
I20260806 03:05:43.902483 23014 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2270660ab23436c9d79e552d4e188bf: Neither blocks nor log segments found. Creating new log.
I20260806 03:05:43.903192 32043 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 03:05:43.904170 23014 log.cc:824] T 00000000000000000000000000000000 P a2270660ab23436c9d79e552d4e188bf: Log is configured to *not* fsync() on all Append() calls
I20260806 03:05:43.905825 22937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74a766d5df9a455f839d7667a558efeb: Bootstrap starting.
I20260806 03:05:43.908005 23014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2270660ab23436c9d79e552d4e188bf: No bootstrap required, opened a new log
I20260806 03:05:43.910617 22937 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 74a766d5df9a455f839d7667a558efeb: Neither blocks nor log segments found. Creating new log.
I20260806 03:05:43.907629 22835 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "63ab3bf44b344bfd8d2c047862544cd2"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45716
I20260806 03:05:43.912206 22937 log.cc:824] T 00000000000000000000000000000000 P 74a766d5df9a455f839d7667a558efeb: Log is configured to *not* fsync() on all Append() calls
I20260806 03:05:43.916853 22937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74a766d5df9a455f839d7667a558efeb: No bootstrap required, opened a new log
I20260806 03:05:43.919083 22860 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.74.252" port: 41949 } attempt: 5
I20260806 03:05:43.920536 23014 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2270660ab23436c9d79e552d4e188bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63ab3bf44b344bfd8d2c047862544cd2" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 37453 } } peers { permanent_uuid: "74a766d5df9a455f839d7667a558efeb" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 37047 } } peers { permanent_uuid: "a2270660ab23436c9d79e552d4e188bf" member_type: VOTER last_known_addr { host: "127.31.74.252" port: 41949 } }
I20260806 03:05:43.921950 23014 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2270660ab23436c9d79e552d4e188bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2270660ab23436c9d79e552d4e188bf, State: Initialized, Role: FOLLOWER
I20260806 03:05:43.922955 23014 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2270660ab23436c9d79e552d4e188bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63ab3bf44b344bfd8d2c047862544cd2" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 37453 } } peers { permanent_uuid: "74a766d5df9a455f839d7667a558efeb" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 37047 } } peers { permanent_uuid: "a2270660ab23436c9d79e552d4e188bf" member_type: VOTER last_known_addr { host: "127.31.74.252" port: 41949 } }
I20260806 03:05:43.926682 23024 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2270660ab23436c9d79e552d4e188bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63ab3bf44b344bfd8d2c047862544cd2" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 37453 } } peers { permanent_uuid: "74a766d5df9a455f839d7667a558efeb" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 37047 } } peers { permanent_uuid: "a2270660ab23436c9d79e552d4e188bf" member_type: VOTER last_known_addr { host: "127.31.74.252" port: 41949 } } }
I20260806 03:05:43.927879 23024 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2270660ab23436c9d79e552d4e188bf [sys.catalog]: This master's current role is: FOLLOWER
I20260806 03:05:43.929420 23014 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2270660ab23436c9d79e552d4e188bf [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:05:43.933668 22937 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74a766d5df9a455f839d7667a558efeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63ab3bf44b344bfd8d2c047862544cd2" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 37453 } } peers { permanent_uuid: "74a766d5df9a455f839d7667a558efeb" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 37047 } } peers { permanent_uuid: "a2270660ab23436c9d79e552d4e188bf" member_type: VOTER last_known_addr { host: "127.31.74.252" port: 41949 } }
I20260806 03:05:43.934695 22937 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74a766d5df9a455f839d7667a558efeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74a766d5df9a455f839d7667a558efeb, State: Initialized, Role: FOLLOWER
Aug 06 03:05:43 dist-test-slave-dhph krb5kdc[22775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985542, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.254@KRBTEST.COM for kudu/127.31.74.252@KRBTEST.COM
I20260806 03:05:43.935561 22937 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74a766d5df9a455f839d7667a558efeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63ab3bf44b344bfd8d2c047862544cd2" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 37453 } } peers { permanent_uuid: "74a766d5df9a455f839d7667a558efeb" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 37047 } } peers { permanent_uuid: "a2270660ab23436c9d79e552d4e188bf" member_type: VOTER last_known_addr { host: "127.31.74.252" port: 41949 } }
I20260806 03:05:43.939886 23026 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74a766d5df9a455f839d7667a558efeb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63ab3bf44b344bfd8d2c047862544cd2" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 37453 } } peers { permanent_uuid: "74a766d5df9a455f839d7667a558efeb" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 37047 } } peers { permanent_uuid: "a2270660ab23436c9d79e552d4e188bf" member_type: VOTER last_known_addr { host: "127.31.74.252" port: 41949 } } }
I20260806 03:05:43.940382 23026 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74a766d5df9a455f839d7667a558efeb [sys.catalog]: This master's current role is: FOLLOWER
I20260806 03:05:43.941870 22937 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74a766d5df9a455f839d7667a558efeb [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:05:43.945634 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:43.919889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.252:41949 (local address 127.0.0.1:58148)
0806 03:05:43.920435 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0806 03:05:43.920459 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 03:05:43.920570 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:05:43.923486 (+ 2916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:05:43.923493 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:05:43.925009 (+ 1516us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:05:43.925530 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:05:43.925550 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:05:43.927524 (+ 1974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:05:43.927531 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:05:43.929107 (+ 1576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:05:43.929134 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:05:43.930511 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:05:43.930521 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:05:43.930838 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:05:43.931583 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:05:43.931608 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:05:43.935871 (+ 4263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:05:43.941700 (+ 5829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:05:43.941710 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:05:43.941717 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:05:43.942437 (+ 720us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:05:43.943150 (+ 713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:05:43.943159 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:05:43.943165 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:05:43.943459 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:05:43.944424 (+ 965us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:05:43.944432 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:05:43.944612 (+ 180us) client_negotiation.cc:769] Sending connection context
0806 03:05:43.944783 (+ 171us) client_negotiation.cc:240] Negotiation successful
0806 03:05:43.945152 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233}
I20260806 03:05:43.946672 23021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:43.919966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58148 (local address 127.31.74.252:41949)
0806 03:05:43.921956 (+ 1990us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:43.921965 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:43.921997 (+ 32us) server_negotiation.cc:407] Connection header received
0806 03:05:43.922072 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:43.922077 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:43.922343 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:43.923196 (+ 853us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:43.925810 (+ 2614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:43.927286 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:43.929338 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:43.930105 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:43.930414 (+ 309us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:05:43.936270 (+ 5856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:05:43.936279 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:05:43.936285 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:05:43.936338 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:05:43.941450 (+ 5112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:05:43.942623 (+ 1173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:05:43.942630 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:05:43.942634 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:05:43.942838 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:05:43.943632 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:05:43.943640 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:05:43.943647 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:05:43.944148 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:05:43.944309 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0806 03:05:43.945050 (+ 741us) server_negotiation.cc:299] Negotiation successful
0806 03:05:43.945517 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":690}
I20260806 03:05:43.956099 22860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63ab3bf44b344bfd8d2c047862544cd2: Bootstrap starting.
I20260806 03:05:43.961179 22860 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63ab3bf44b344bfd8d2c047862544cd2: Neither blocks nor log segments found. Creating new log.
I20260806 03:05:43.962790 22860 log.cc:824] T 00000000000000000000000000000000 P 63ab3bf44b344bfd8d2c047862544cd2: Log is configured to *not* fsync() on all Append() calls
I20260806 03:05:43.968357 22860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63ab3bf44b344bfd8d2c047862544cd2: No bootstrap required, opened a new log
I20260806 03:05:43.981684 22860 raft_consensus.cc:348] T 00000000000000000000000000000000 P 63ab3bf44b344bfd8d2c047862544cd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63ab3bf44b344bfd8d2c047862544cd2" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 37453 } } peers { permanent_uuid: "74a766d5df9a455f839d7667a558efeb" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 37047 } } peers { permanent_uuid: "a2270660ab23436c9d79e552d4e188bf" member_type: VOTER last_known_addr { host: "127.31.74.252" port: 41949 } }
I20260806 03:05:43.982611 22860 raft_consensus.cc:729] T 00000000000000000000000000000000 P 63ab3bf44b344bfd8d2c047862544cd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63ab3bf44b344bfd8d2c047862544cd2, State: Initialized, Role: FOLLOWER
I20260806 03:05:43.983752 22860 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63ab3bf44b344bfd8d2c047862544cd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63ab3bf44b344bfd8d2c047862544cd2" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 37453 } } peers { permanent_uuid: "74a766d5df9a455f839d7667a558efeb" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 37047 } } peers { permanent_uuid: "a2270660ab23436c9d79e552d4e188bf" member_type: VOTER last_known_addr { host: "127.31.74.252" port: 41949 } }
W20260806 03:05:43.986336 23048 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a2270660ab23436c9d79e552d4e188bf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 03:05:43.986660 23048 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 03:05:43.987581 23051 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 03:05:43.976315 22946 master_runner.cc:186] Error getting master registration for 127.31.74.254:37453: OK, instance_id { permanent_uuid: "63ab3bf44b344bfd8d2c047862544cd2" instance_seqno: 1785985542459659 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 03:05:44.000801 22946 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 03:05:44.001787 22860 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63ab3bf44b344bfd8d2c047862544cd2 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:05:43.990543 22867 master_runner.cc:186] Error getting master registration for 127.31.74.254:37453: OK, instance_id { permanent_uuid: "63ab3bf44b344bfd8d2c047862544cd2" instance_seqno: 1785985542459659 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 03:05:44.002796 22835 raft_consensus.cc:482] T 00000000000000000000000000000000 P 63ab3bf44b344bfd8d2c047862544cd2 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:05:44.003070 22867 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 03:05:44.003057 22835 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 63ab3bf44b344bfd8d2c047862544cd2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:05:44.006733 23052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63ab3bf44b344bfd8d2c047862544cd2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63ab3bf44b344bfd8d2c047862544cd2" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 37453 } } peers { permanent_uuid: "74a766d5df9a455f839d7667a558efeb" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 37047 } } peers { permanent_uuid: "a2270660ab23436c9d79e552d4e188bf" member_type: VOTER last_known_addr { host: "127.31.74.252" port: 41949 } } }
I20260806 03:05:44.007424 23052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63ab3bf44b344bfd8d2c047862544cd2 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 03:05:44.008462 22835 raft_consensus.cc:504] T 00000000000000000000000000000000 P 63ab3bf44b344bfd8d2c047862544cd2 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63ab3bf44b344bfd8d2c047862544cd2" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 37453 } } peers { permanent_uuid: "74a766d5df9a455f839d7667a558efeb" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 37047 } } peers { permanent_uuid: "a2270660ab23436c9d79e552d4e188bf" member_type: VOTER last_known_addr { host: "127.31.74.252" port: 41949 } }
I20260806 03:05:44.012316 22835 leader_election.cc:288] T 00000000000000000000000000000000 P 63ab3bf44b344bfd8d2c047862544cd2 [CANDIDATE]: Term 1 election: Requested vote from peers 74a766d5df9a455f839d7667a558efeb (127.31.74.253:37047), a2270660ab23436c9d79e552d4e188bf (127.31.74.252:41949)
I20260806 03:05:44.012989 22989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "63ab3bf44b344bfd8d2c047862544cd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a2270660ab23436c9d79e552d4e188bf"
I20260806 03:05:44.013074 22911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "63ab3bf44b344bfd8d2c047862544cd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "74a766d5df9a455f839d7667a558efeb"
I20260806 03:05:44.013622 22989 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2270660ab23436c9d79e552d4e188bf [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:05:44.013675 22911 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74a766d5df9a455f839d7667a558efeb [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:05:44.017684 22911 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 74a766d5df9a455f839d7667a558efeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63ab3bf44b344bfd8d2c047862544cd2 in term 1.
I20260806 03:05:44.018729 22809 leader_election.cc:302] T 00000000000000000000000000000000 P 63ab3bf44b344bfd8d2c047862544cd2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63ab3bf44b344bfd8d2c047862544cd2, 74a766d5df9a455f839d7667a558efeb; no voters:
I20260806 03:05:44.018905 22989 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a2270660ab23436c9d79e552d4e188bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63ab3bf44b344bfd8d2c047862544cd2 in term 1.
I20260806 03:05:44.019578 23052 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 63ab3bf44b344bfd8d2c047862544cd2 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:05:44.029718 23052 raft_consensus.cc:686] T 00000000000000000000000000000000 P 63ab3bf44b344bfd8d2c047862544cd2 [term 1 LEADER]: Becoming Leader. State: Replica: 63ab3bf44b344bfd8d2c047862544cd2, State: Running, Role: LEADER
I20260806 03:05:44.030737 23052 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63ab3bf44b344bfd8d2c047862544cd2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63ab3bf44b344bfd8d2c047862544cd2" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 37453 } } peers { permanent_uuid: "74a766d5df9a455f839d7667a558efeb" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 37047 } } peers { permanent_uuid: "a2270660ab23436c9d79e552d4e188bf" member_type: VOTER last_known_addr { host: "127.31.74.252" port: 41949 } }
W20260806 03:05:44.033521 22968 tablet.cc:2369] T 00000000000000000000000000000000 P a2270660ab23436c9d79e552d4e188bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 03:05:44.042760 23060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63ab3bf44b344bfd8d2c047862544cd2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63ab3bf44b344bfd8d2c047862544cd2. Latest consensus state: current_term: 1 leader_uuid: "63ab3bf44b344bfd8d2c047862544cd2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63ab3bf44b344bfd8d2c047862544cd2" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 37453 } } peers { permanent_uuid: "74a766d5df9a455f839d7667a558efeb" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 37047 } } peers { permanent_uuid: "a2270660ab23436c9d79e552d4e188bf" member_type: VOTER last_known_addr { host: "127.31.74.252" port: 41949 } } }
I20260806 03:05:44.043488 23060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63ab3bf44b344bfd8d2c047862544cd2 [sys.catalog]: This master's current role is: LEADER
I20260806 03:05:44.053490 23070 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:05:44.063169 23069 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 03:05:44.072175 23021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:44.047012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58164 (local address 127.31.74.252:41949)
0806 03:05:44.047246 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:44.047254 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:44.047282 (+ 28us) server_negotiation.cc:407] Connection header received
0806 03:05:44.047593 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:44.047602 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:44.047703 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:44.047787 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:44.050753 (+ 2966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:44.052656 (+ 1903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:44.055458 (+ 2802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:44.056378 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:44.056577 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:05:44.061419 (+ 4842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:05:44.061430 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:05:44.061437 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:05:44.061491 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:05:44.066162 (+ 4671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:05:44.068241 (+ 2079us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:05:44.068250 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:05:44.068256 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:05:44.068554 (+ 298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:05:44.069161 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:05:44.069167 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:05:44.069171 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:05:44.069596 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:05:44.070006 (+ 410us) server_negotiation.cc:1035] Waiting for connection context
0806 03:05:44.071084 (+ 1078us) server_negotiation.cc:299] Negotiation successful
0806 03:05:44.071505 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260806 03:05:44.078933 23070 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:05:44.096297 22989 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a2270660ab23436c9d79e552d4e188bf [term 1 FOLLOWER]: Refusing update from remote peer 63ab3bf44b344bfd8d2c047862544cd2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:05:44.097914 23060 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 63ab3bf44b344bfd8d2c047862544cd2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2270660ab23436c9d79e552d4e188bf" member_type: VOTER last_known_addr { host: "127.31.74.252" port: 41949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 03:05:44.100905 22888 tablet.cc:2369] T 00000000000000000000000000000000 P 74a766d5df9a455f839d7667a558efeb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 03:05:44.105741 22911 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 74a766d5df9a455f839d7667a558efeb [term 1 FOLLOWER]: Refusing update from remote peer 63ab3bf44b344bfd8d2c047862544cd2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:05:44.106078 23020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:44.047877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35898 (local address 127.31.74.253:37047)
0806 03:05:44.048061 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:44.048066 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:44.057277 (+ 9211us) server_negotiation.cc:407] Connection header received
0806 03:05:44.060021 (+ 2744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:44.060031 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:44.060137 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:44.060237 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:44.066057 (+ 5820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:44.067585 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:44.076560 (+ 8975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:44.077326 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:44.077475 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:05:44.091465 (+ 13990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:05:44.091474 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:05:44.091479 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:05:44.091545 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:05:44.095363 (+ 3818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:05:44.096428 (+ 1065us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:05:44.096433 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:05:44.096437 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:05:44.096628 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:05:44.103231 (+ 6603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:05:44.103240 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:05:44.103244 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:05:44.103760 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:05:44.104090 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0806 03:05:44.105234 (+ 1144us) server_negotiation.cc:299] Negotiation successful
0806 03:05:44.105542 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 03:05:44.106876 23060 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 63ab3bf44b344bfd8d2c047862544cd2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a766d5df9a455f839d7667a558efeb" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:05:44.119678 23018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:44.045973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45724 (local address 127.31.74.254:37453)
0806 03:05:44.046183 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:44.046191 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:44.059387 (+ 13196us) server_negotiation.cc:407] Connection header received
0806 03:05:44.063524 (+ 4137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:44.063533 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:44.063633 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:44.063736 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:44.069989 (+ 6253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:44.071478 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:44.079933 (+ 8455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:44.080624 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:44.080727 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:05:44.097585 (+ 16858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:05:44.097594 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:05:44.097599 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:05:44.097653 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:05:44.102036 (+ 4383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:05:44.114964 (+ 12928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:05:44.114971 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:05:44.114976 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:05:44.115216 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:05:44.116700 (+ 1484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:05:44.116706 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:05:44.116710 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:05:44.117227 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:05:44.117738 (+ 511us) server_negotiation.cc:1035] Waiting for connection context
0806 03:05:44.118886 (+ 1148us) server_negotiation.cc:299] Negotiation successful
0806 03:05:44.119234 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 03:05:44.134804 23024 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2270660ab23436c9d79e552d4e188bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63ab3bf44b344bfd8d2c047862544cd2. Latest consensus state: current_term: 1 leader_uuid: "63ab3bf44b344bfd8d2c047862544cd2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63ab3bf44b344bfd8d2c047862544cd2" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 37453 } } peers { permanent_uuid: "74a766d5df9a455f839d7667a558efeb" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 37047 } } peers { permanent_uuid: "a2270660ab23436c9d79e552d4e188bf" member_type: VOTER last_known_addr { host: "127.31.74.252" port: 41949 } } }
I20260806 03:05:44.135254 23024 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2270660ab23436c9d79e552d4e188bf [sys.catalog]: This master's current role is: FOLLOWER
I20260806 03:05:44.143836 23075 mvcc.cc:205] Tried to move back new op lower bound from 7315396788602683392 to 7315396788357865472. Current Snapshot: MvccSnapshot[applied={T|T < 7315396788602683392}]
I20260806 03:05:44.143802 23026 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74a766d5df9a455f839d7667a558efeb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63ab3bf44b344bfd8d2c047862544cd2. Latest consensus state: current_term: 1 leader_uuid: "63ab3bf44b344bfd8d2c047862544cd2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63ab3bf44b344bfd8d2c047862544cd2" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 37453 } } peers { permanent_uuid: "74a766d5df9a455f839d7667a558efeb" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 37047 } } peers { permanent_uuid: "a2270660ab23436c9d79e552d4e188bf" member_type: VOTER last_known_addr { host: "127.31.74.252" port: 41949 } } }
I20260806 03:05:44.144158 23026 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74a766d5df9a455f839d7667a558efeb [sys.catalog]: This master's current role is: FOLLOWER
I20260806 03:05:44.150092 23076 mvcc.cc:205] Tried to move back new op lower bound from 7315396788602683392 to 7315396788357865472. Current Snapshot: MvccSnapshot[applied={T|T < 7315396788602683392}]
I20260806 03:05:44.150322 23026 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74a766d5df9a455f839d7667a558efeb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "63ab3bf44b344bfd8d2c047862544cd2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63ab3bf44b344bfd8d2c047862544cd2" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 37453 } } peers { permanent_uuid: "74a766d5df9a455f839d7667a558efeb" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 37047 } } peers { permanent_uuid: "a2270660ab23436c9d79e552d4e188bf" member_type: VOTER last_known_addr { host: "127.31.74.252" port: 41949 } } }
I20260806 03:05:44.150765 23026 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74a766d5df9a455f839d7667a558efeb [sys.catalog]: This master's current role is: FOLLOWER
I20260806 03:05:44.156004 23060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63ab3bf44b344bfd8d2c047862544cd2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "63ab3bf44b344bfd8d2c047862544cd2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63ab3bf44b344bfd8d2c047862544cd2" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 37453 } } peers { permanent_uuid: "74a766d5df9a455f839d7667a558efeb" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 37047 } } peers { permanent_uuid: "a2270660ab23436c9d79e552d4e188bf" member_type: VOTER last_known_addr { host: "127.31.74.252" port: 41949 } } }
I20260806 03:05:44.156450 23060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63ab3bf44b344bfd8d2c047862544cd2 [sys.catalog]: This master's current role is: LEADER
I20260806 03:05:44.156725 23060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63ab3bf44b344bfd8d2c047862544cd2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "63ab3bf44b344bfd8d2c047862544cd2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63ab3bf44b344bfd8d2c047862544cd2" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 37453 } } peers { permanent_uuid: "74a766d5df9a455f839d7667a558efeb" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 37047 } } peers { permanent_uuid: "a2270660ab23436c9d79e552d4e188bf" member_type: VOTER last_known_addr { host: "127.31.74.252" port: 41949 } } }
I20260806 03:05:44.157058 23060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63ab3bf44b344bfd8d2c047862544cd2 [sys.catalog]: This master's current role is: LEADER
I20260806 03:05:44.162952 23024 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2270660ab23436c9d79e552d4e188bf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "63ab3bf44b344bfd8d2c047862544cd2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63ab3bf44b344bfd8d2c047862544cd2" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 37453 } } peers { permanent_uuid: "74a766d5df9a455f839d7667a558efeb" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 37047 } } peers { permanent_uuid: "a2270660ab23436c9d79e552d4e188bf" member_type: VOTER last_known_addr { host: "127.31.74.252" port: 41949 } } }
I20260806 03:05:44.163367 23024 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2270660ab23436c9d79e552d4e188bf [sys.catalog]: This master's current role is: FOLLOWER
I20260806 03:05:44.165346 23070 catalog_manager.cc:1347] Generated new cluster ID: 4f997b42464442c29b828325c185bb51
I20260806 03:05:44.165472 23070 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:05:44.195541 23070 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:05:44.197810 23070 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:05:44.218073 23070 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 63ab3bf44b344bfd8d2c047862544cd2: Generated new TSK 0
I20260806 03:05:44.219473 23070 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 03:05:44.328548 23020 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:05:44.307517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35904 (local address 127.31.74.253:37047)
0806 03:05:44.307822 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:44.307831 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:44.324005 (+ 16174us) server_negotiation.cc:407] Connection header received
0806 03:05:44.324402 (+ 397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:44.324409 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:44.324525 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:44.324898 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:44.328114 (+ 3216us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35904: BlockingRecv error: recv got EOF from 127.0.0.1:35904 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
W20260806 03:05:44.329170 23021 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:05:44.310545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58180 (local address 127.31.74.252:41949)
0806 03:05:44.310813 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:44.310823 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:44.326225 (+ 15402us) server_negotiation.cc:407] Connection header received
0806 03:05:44.326461 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:44.326468 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:44.326617 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:44.326749 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:44.328921 (+ 2172us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58180: BlockingRecv error: recv got EOF from 127.0.0.1:58180 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260806 03:05:44.331159 23018 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:05:44.306981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45736 (local address 127.31.74.254:37453)
0806 03:05:44.307301 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:44.307309 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:44.324004 (+ 16695us) server_negotiation.cc:407] Connection header received
0806 03:05:44.324389 (+ 385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:44.324397 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:44.324515 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:44.324898 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:44.330848 (+ 5950us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45736: BlockingRecv error: recv got EOF from 127.0.0.1:45736 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
W20260806 03:05:44.344857 23020 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:05:44.338013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35912 (local address 127.31.74.253:37047)
0806 03:05:44.338280 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:44.338291 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:44.342853 (+ 4562us) server_negotiation.cc:407] Connection header received
0806 03:05:44.343170 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:44.343176 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:44.343303 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:44.343740 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:44.344664 (+ 924us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35912: BlockingRecv error: recv got EOF from 127.0.0.1:35912 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260806 03:05:44.359714 23020 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:05:44.357105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35916 (local address 127.31.74.253:37047)
0806 03:05:44.357372 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:44.357382 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:44.357950 (+ 568us) server_negotiation.cc:407] Connection header received
0806 03:05:44.358199 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:44.358206 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:44.358297 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:44.358428 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:44.359436 (+ 1008us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35916: BlockingRecv error: recv got EOF from 127.0.0.1:35916 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260806 03:05:44.368705 23020 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:05:44.365840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35926 (local address 127.31.74.253:37047)
0806 03:05:44.366410 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:44.366422 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:44.367030 (+ 608us) server_negotiation.cc:407] Connection header received
0806 03:05:44.367280 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:44.367287 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:44.367383 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:44.367513 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:44.368471 (+ 958us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35926: BlockingRecv error: recv got EOF from 127.0.0.1:35926 (error 108)
Metrics: {"server-negotiator.queue_time_us":339}
W20260806 03:05:44.380188 23020 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:05:44.377227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35930 (local address 127.31.74.253:37047)
0806 03:05:44.377598 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:44.377609 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:44.378215 (+ 606us) server_negotiation.cc:407] Connection header received
0806 03:05:44.378537 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:44.378543 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:44.378694 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:44.378819 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:44.379942 (+ 1123us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35930: BlockingRecv error: recv got EOF from 127.0.0.1:35930 (error 108)
Metrics: {"server-negotiator.queue_time_us":132}
W20260806 03:05:44.396148 23020 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:05:44.393462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35946 (local address 127.31.74.253:37047)
0806 03:05:44.393737 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:44.393748 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:44.394468 (+ 720us) server_negotiation.cc:407] Connection header received
0806 03:05:44.394861 (+ 393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:44.394867 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:44.394975 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:44.395092 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:44.395884 (+ 792us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35946: BlockingRecv error: recv got EOF from 127.0.0.1:35946 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
W20260806 03:05:44.419724 23020 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:05:44.416868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35952 (local address 127.31.74.253:37047)
0806 03:05:44.417112 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:44.417123 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:44.417845 (+ 722us) server_negotiation.cc:407] Connection header received
0806 03:05:44.418244 (+ 399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:44.418258 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:44.418341 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:44.418468 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:44.419452 (+ 984us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35952: BlockingRecv error: recv got EOF from 127.0.0.1:35952 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260806 03:05:44.460770 23020 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:05:44.458101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35962 (local address 127.31.74.253:37047)
0806 03:05:44.458380 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:44.458391 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:44.459016 (+ 625us) server_negotiation.cc:407] Connection header received
0806 03:05:44.459338 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:44.459348 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:44.459482 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:44.459614 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:44.460493 (+ 879us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35962: BlockingRecv error: recv got EOF from 127.0.0.1:35962 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
W20260806 03:05:44.537072 23020 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:05:44.532882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35966 (local address 127.31.74.253:37047)
0806 03:05:44.533186 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:44.533196 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:44.533857 (+ 661us) server_negotiation.cc:407] Connection header received
0806 03:05:44.534375 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:44.534388 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:44.534570 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:44.534799 (+ 229us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:44.536761 (+ 1962us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35966: BlockingRecv error: recv got EOF from 127.0.0.1:35966 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
W20260806 03:05:44.675313 23020 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:05:44.672527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35970 (local address 127.31.74.253:37047)
0806 03:05:44.672819 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:44.672831 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:44.673505 (+ 674us) server_negotiation.cc:407] Connection header received
0806 03:05:44.673846 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:44.673851 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:44.673938 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:44.674070 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:44.675015 (+ 945us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35970: BlockingRecv error: recv got EOF from 127.0.0.1:35970 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
W20260806 03:05:44.938755 23020 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:05:44.936488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35984 (local address 127.31.74.253:37047)
0806 03:05:44.936686 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:44.936692 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:44.937054 (+ 362us) server_negotiation.cc:407] Connection header received
0806 03:05:44.937318 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:44.937322 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:44.937418 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:44.937518 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:44.938440 (+ 922us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35984: BlockingRecv error: recv got EOF from 127.0.0.1:35984 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260806 03:05:44.989339 23048 catalog_manager.cc:1259] Loaded cluster ID: 4f997b42464442c29b828325c185bb51
I20260806 03:05:44.989450 23048 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a2270660ab23436c9d79e552d4e188bf: loading cluster ID for follower catalog manager: success
I20260806 03:05:44.992262 23048 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a2270660ab23436c9d79e552d4e188bf: acquiring CA information for follower catalog manager: success
I20260806 03:05:44.993279 23051 catalog_manager.cc:1259] Loaded cluster ID: 4f997b42464442c29b828325c185bb51
I20260806 03:05:44.993407 23051 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 74a766d5df9a455f839d7667a558efeb: loading cluster ID for follower catalog manager: success
I20260806 03:05:44.994344 23048 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a2270660ab23436c9d79e552d4e188bf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 03:05:44.997208 23051 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 74a766d5df9a455f839d7667a558efeb: acquiring CA information for follower catalog manager: success
I20260806 03:05:44.999903 23051 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 74a766d5df9a455f839d7667a558efeb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 03:05:45.470340 23153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:45.460718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36000 (local address 127.31.74.253:37047)
0806 03:05:45.463408 (+ 2690us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:45.463415 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:45.463439 (+ 24us) server_negotiation.cc:407] Connection header received
0806 03:05:45.463520 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:45.463525 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:45.463770 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:45.463945 (+ 175us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 03:05:45.464506 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:45.465433 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:45.467503 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:45.468033 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:45.468217 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:05:45.468888 (+ 671us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 03:05:45.469478 (+ 590us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 03:05:45.469607 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0806 03:05:45.469761 (+ 154us) server_negotiation.cc:299] Negotiation successful
0806 03:05:45.469842 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2534,"thread_start_us":2178,"threads_started":1}
I20260806 03:05:45.487486 23159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:45.479001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58186 (local address 127.31.74.252:41949)
0806 03:05:45.479684 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:45.479694 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:45.479847 (+ 153us) server_negotiation.cc:407] Connection header received
0806 03:05:45.480260 (+ 413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:45.480265 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:45.480607 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:45.480876 (+ 269us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 03:05:45.481633 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:45.482870 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:45.484515 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:45.484989 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:45.485155 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:05:45.485737 (+ 582us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 03:05:45.486726 (+ 989us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 03:05:45.486936 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0806 03:05:45.487005 (+ 69us) server_negotiation.cc:299] Negotiation successful
0806 03:05:45.487107 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":310,"threads_started":1}
I20260806 03:05:45.521751 23159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:45.497027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58202 (local address 127.31.74.252:41949)
0806 03:05:45.497409 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:45.497419 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:45.512274 (+ 14855us) server_negotiation.cc:407] Connection header received
0806 03:05:45.512715 (+ 441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:45.512721 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:45.512861 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:45.512980 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 03:05:45.513892 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:45.515246 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:45.519382 (+ 4136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:45.519939 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:45.520105 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:05:45.520816 (+ 711us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 03:05:45.520979 (+ 163us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 03:05:45.521073 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0806 03:05:45.521300 (+ 227us) server_negotiation.cc:299] Negotiation successful
0806 03:05:45.521394 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260806 03:05:45.522017 23153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:45.496412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36008 (local address 127.31.74.253:37047)
0806 03:05:45.496666 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:45.496677 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:45.513654 (+ 16977us) server_negotiation.cc:407] Connection header received
0806 03:05:45.514393 (+ 739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:45.514400 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:45.514504 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:45.514637 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 03:05:45.515494 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:45.516283 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:45.519176 (+ 2893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:45.519834 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:45.520063 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:05:45.521055 (+ 992us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 03:05:45.521195 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 03:05:45.521307 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0806 03:05:45.521473 (+ 166us) server_negotiation.cc:299] Negotiation successful
0806 03:05:45.521587 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 03:05:45.523077 23165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:45.496246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45744 (local address 127.31.74.254:37453)
0806 03:05:45.510515 (+ 14269us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:45.510526 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:45.511841 (+ 1315us) server_negotiation.cc:407] Connection header received
0806 03:05:45.512383 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:45.512392 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:45.512586 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:45.512828 (+ 242us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 03:05:45.513919 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:45.515035 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:45.518210 (+ 3175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:45.519113 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:45.519459 (+ 346us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:05:45.521142 (+ 1683us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 03:05:45.521652 (+ 510us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 03:05:45.522063 (+ 411us) server_negotiation.cc:1035] Waiting for connection context
0806 03:05:45.522494 (+ 431us) server_negotiation.cc:299] Negotiation successful
0806 03:05:45.522660 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14034,"thread_start_us":15632,"threads_started":1}
I20260806 03:05:45.531764 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 22789
I20260806 03:05:45.533008 22855 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 03:05:45.663020 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 22789
I20260806 03:05:45.692907 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 22867
I20260806 03:05:45.694275 22932 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 03:05:45.830277 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 22867
I20260806 03:05:45.864116 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 22946
I20260806 03:05:45.865820 23009 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 03:05:46.000906 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 22946
2026-08-06T03:05:46Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (6088 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785985105255475-32043-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 03:05:46 dist-test-slave-dhph krb5kdc[23174](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 03:05:46 dist-test-slave-dhph krb5kdc[23174](info): set up 2 sockets
Aug 06 03:05:46 dist-test-slave-dhph krb5kdc[23174](info): commencing operation
krb5kdc: starting...
W20260806 03:05:47.407547 32043 mini_kdc.cc:121] Time spent starting KDC: real 1.349s user 0.004s sys 0.071s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785985105255475-32043-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785985105255475-32043-0/krb5kdc/test-user.keytab.
Aug 06 03:05:47 dist-test-slave-dhph krb5kdc[23174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T03:05:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T03:05:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.74.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab.
Entry for principal kudu/127.31.74.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab.
I20260806 03:05:47.836441 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785985105255475-32043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785985105255475-32043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785985105255475-32043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:37433
--webserver_interface=127.31.74.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab
--principal=kudu/127.31.74.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:43523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.74.254:37433,127.31.74.253:38467,127.31.74.252:41723
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 03:05:48.273969 23187 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 03:05:48.274359 23187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:05:48.274523 23187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:05:48.274712 23187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:05:48.282512 23187 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:05:48.282687 23187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:05:48.282768 23187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:05:48.282833 23187 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:05:48.282910 23187 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:05:48.294128 23187 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:43523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785985105255475-32043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785985105255475-32043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.74.254:37433,127.31.74.253:38467,127.31.74.252:41723
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.254.keytab
--principal=kudu/127.31.74.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.254:37433
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.74.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785985105255475-32043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:05:48.295758 23187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:05:48.297175 23187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:05:48.311354 23192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:05:48.311292 23193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:05:48.311272 23195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:05:48.312515 23187 server_base.cc:1034] running on GCE node
I20260806 03:05:48.313526 23187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:05:48.315263 23187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:05:48.316569 23187 hybrid_clock.cc:648] HybridClock initialized: now 1785985548316477 us; error 73 us; skew 500 ppm
I20260806 03:05:48.316990 23187 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 03:05:48 dist-test-slave-dhph krb5kdc[23174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985548, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:05:48.321059 23187 init.cc:377] Logged in from keytab as kudu/127.31.74.254@KRBTEST.COM (short username kudu)
I20260806 03:05:48.322983 23187 webserver.cc:460] Webserver started at http://127.31.74.254:38795/ using document root <none> and password file <none>
I20260806 03:05:48.323712 23187 fs_manager.cc:359] Metadata directory not provided
I20260806 03:05:48.323808 23187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:05:48.324080 23187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:05:48.327333 23187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785985105255475-32043-0/minicluster-data/master-0/data/instance:
uuid: "e99b8e574f924612ab48cc2a396a169a"
format_stamp: "Formatted at 2026-08-06 03:05:48 on dist-test-slave-dhph"
server_key: "ea298b8dbfcc89cdaf1a61b2c7274091"
server_key_iv: "743dd9547b45482f36f52da91d8cae9b"
server_key_version: "encryptionkey@0"
I20260806 03:05:48.328157 23187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785985105255475-32043-0/minicluster-data/master-0/wal/instance:
uuid: "e99b8e574f924612ab48cc2a396a169a"
format_stamp: "Formatted at 2026-08-06 03:05:48 on dist-test-slave-dhph"
server_key: "ea298b8dbfcc89cdaf1a61b2c7274091"
server_key_iv: "743dd9547b45482f36f52da91d8cae9b"
server_key_version: "encryptionkey@0"
I20260806 03:05:48.334553 23187 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.006s
I20260806 03:05:48.338670 23202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:05:48.340456 23187 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260806 03:05:48.340659 23187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785985105255475-32043-0/minicluster-data/master-0/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785985105255475-32043-0/minicluster-data/master-0/wal
uuid: "e99b8e574f924612ab48cc2a396a169a"
format_stamp: "Formatted at 2026-08-06 03:05:48 on dist-test-slave-dhph"
server_key: "ea298b8dbfcc89cdaf1a61b2c7274091"
server_key_iv: "743dd9547b45482f36f52da91d8cae9b"
server_key_version: "encryptionkey@0"
I20260806 03:05:48.340899 23187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785985105255475-32043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785985105255475-32043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785985105255475-32043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:05:48.375334 23187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:05:48.380116 23187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:05:48.380374 23187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:05:48.401901 23187 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.254:37433
I20260806 03:05:48.401943 23255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.254:37433 every 8 connection(s)
I20260806 03:05:48.403364 23187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785985105255475-32043-0/minicluster-data/master-0/data/info.pb
I20260806 03:05:48.407405 23256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:05:48.412880 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 23187
I20260806 03:05:48.413096 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785985105255475-32043-0/minicluster-data/master-0/wal/instance
I20260806 03:05:48.413650 32043 external_mini_cluster.cc:1428] Setting key c003a1a795e6a3e785304b98ed0d6abb
I20260806 03:05:48.416023 23256 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.74.254" port: 37433 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 03:05:48 dist-test-slave-dhph krb5kdc[23174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985548, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.254@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 03:05:48.447463 23259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:48.421819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54700 (local address 127.31.74.254:37433)
0806 03:05:48.423704 (+ 1885us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:48.423719 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:48.423788 (+ 69us) server_negotiation.cc:407] Connection header received
0806 03:05:48.425273 (+ 1485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:48.425284 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:48.425635 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:48.425873 (+ 238us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:48.427287 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:48.429183 (+ 1896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:48.430926 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:48.431930 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:48.432141 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:05:48.438055 (+ 5914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:05:48.438090 (+ 35us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:05:48.438112 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:05:48.438163 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:05:48.441954 (+ 3791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:05:48.443056 (+ 1102us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:05:48.443064 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:05:48.443066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:05:48.443268 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:05:48.443850 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:05:48.443854 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:05:48.443856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:05:48.444472 (+ 616us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:05:48.444646 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0806 03:05:48.445564 (+ 918us) server_negotiation.cc:299] Negotiation successful
0806 03:05:48.445993 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":1359,"thread_start_us":1006,"threads_started":1}
I20260806 03:05:48.447572 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:48.420327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:37433 (local address 127.0.0.1:54700)
0806 03:05:48.421197 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0806 03:05:48.421247 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 03:05:48.424812 (+ 3565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:05:48.425926 (+ 1114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:05:48.425931 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:05:48.426547 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:05:48.427060 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:05:48.427078 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:05:48.429406 (+ 2328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:05:48.429411 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:05:48.430752 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:05:48.430767 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:05:48.432420 (+ 1653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:05:48.432433 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:05:48.432750 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:05:48.433953 (+ 1203us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:05:48.433994 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:05:48.437484 (+ 3490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:05:48.442153 (+ 4669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:05:48.442171 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:05:48.442188 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:05:48.442853 (+ 665us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:05:48.443401 (+ 548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:05:48.443408 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:05:48.443415 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:05:48.443665 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:05:48.444726 (+ 1061us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:05:48.444738 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:05:48.445043 (+ 305us) client_negotiation.cc:769] Sending connection context
0806 03:05:48.445423 (+ 380us) client_negotiation.cc:240] Negotiation successful
0806 03:05:48.446286 (+ 863us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"mutex_wait_us":182,"thread_start_us":388,"threads_started":1}
I20260806 03:05:48.451043 23256 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.74.253" port: 38467 } has no permanent_uuid. Determining permanent_uuid...
I20260806 03:05:48.452484 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:48.451907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.253:38467 (local address 127.0.0.1:57006)
0806 03:05:48.452124 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0806 03:05:48.452303 (+ 179us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.253:38467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
W20260806 03:05:48.452849 23205 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.74.253:38467: connect: Connection refused (error 111)
I20260806 03:05:48.454981 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:48.454458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.253:38467 (local address 127.0.0.1:57022)
0806 03:05:48.454748 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0806 03:05:48.454827 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.253:38467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260806 03:05:48.455236 23256 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.74.253:38467: Network error: Client connection negotiation failed: client connection to 127.31.74.253:38467: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.31.74.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.253@KRBTEST.COM" created.
I20260806 03:05:48.504621 23256 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.74.253" port: 38467 } attempt: 1
I20260806 03:05:48.505991 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:48.505428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.253:38467 (local address 127.0.0.1:57036)
0806 03:05:48.505640 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0806 03:05:48.505841 (+ 201us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.253:38467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260806 03:05:48.507443 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:48.506927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.253:38467 (local address 127.0.0.1:57046)
0806 03:05:48.507199 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0806 03:05:48.507298 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.253:38467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154}
W20260806 03:05:48.507670 23256 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.74.253:38467: Network error: Client connection negotiation failed: client connection to 127.31.74.253:38467: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.253.keytab.
Entry for principal kudu/127.31.74.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.253.keytab.
I20260806 03:05:48.512375 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785985105255475-32043-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785985105255475-32043-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785985105255475-32043-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785985105255475-32043-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.253:38467
--webserver_interface=127.31.74.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.253.keytab
--principal=kudu/127.31.74.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:43523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.74.254:37433,127.31.74.253:38467,127.31.74.252:41723
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 03:05:48.575927 23256 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.74.253" port: 38467 } attempt: 2
I20260806 03:05:48.577337 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:48.576669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.253:38467 (local address 127.0.0.1:57058)
0806 03:05:48.577014 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0806 03:05:48.577140 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.253:38467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260806 03:05:48.578655 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:48.578150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.253:38467 (local address 127.0.0.1:57072)
0806 03:05:48.578344 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0806 03:05:48.578452 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.253:38467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260806 03:05:48.578935 23256 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.74.253:38467: Network error: Client connection negotiation failed: client connection to 127.31.74.253:38467: connect: Connection refused (error 111)
I20260806 03:05:48.670295 23256 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.74.253" port: 38467 } attempt: 3
I20260806 03:05:48.671751 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:48.671149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.253:38467 (local address 127.0.0.1:57080)
0806 03:05:48.671441 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0806 03:05:48.671567 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.253:38467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260806 03:05:48.673028 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:48.672591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.253:38467 (local address 127.0.0.1:57082)
0806 03:05:48.672755 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0806 03:05:48.672847 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.253:38467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260806 03:05:48.673384 23256 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.74.253:38467: Network error: Client connection negotiation failed: client connection to 127.31.74.253:38467: connect: Connection refused (error 111)
I20260806 03:05:48.816740 23256 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.74.253" port: 38467 } attempt: 4
I20260806 03:05:48.818387 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:48.817471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.253:38467 (local address 127.0.0.1:57086)
0806 03:05:48.817860 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0806 03:05:48.817986 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.253:38467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176}
I20260806 03:05:48.820683 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:48.819732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.253:38467 (local address 127.0.0.1:57094)
0806 03:05:48.819926 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0806 03:05:48.820042 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.253:38467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260806 03:05:48.821017 23256 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.74.253:38467: Network error: Client connection negotiation failed: client connection to 127.31.74.253:38467: connect: Connection refused (error 111)
W20260806 03:05:48.942103 23263 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 03:05:48.942502 23263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:05:48.942698 23263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:05:48.942863 23263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:05:48.954130 23263 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:05:48.954530 23263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:05:48.954749 23263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:05:48.954805 23263 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:05:48.954895 23263 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:05:48.967155 23263 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:43523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785985105255475-32043-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785985105255475-32043-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.31.74.254:37433,127.31.74.253:38467,127.31.74.252:41723
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.253.keytab
--principal=kudu/127.31.74.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.253:38467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785985105255475-32043-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.31.74.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785985105255475-32043-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:05:48.968721 23263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:05:48.970193 23263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:05:48.983896 23271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:05:48.983979 23268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:05:48.983940 23269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:05:48.985349 23263 server_base.cc:1034] running on GCE node
I20260806 03:05:48.986850 23263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:05:48.988566 23263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:05:48.990095 23263 hybrid_clock.cc:648] HybridClock initialized: now 1785985548989922 us; error 80 us; skew 500 ppm
I20260806 03:05:48.990742 23263 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 03:05:48 dist-test-slave-dhph krb5kdc[23174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985548, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:05:48.995922 23263 init.cc:377] Logged in from keytab as kudu/127.31.74.253@KRBTEST.COM (short username kudu)
I20260806 03:05:48.998289 23263 webserver.cc:460] Webserver started at http://127.31.74.253:45691/ using document root <none> and password file <none>
I20260806 03:05:48.999212 23263 fs_manager.cc:359] Metadata directory not provided
I20260806 03:05:48.999366 23263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:05:48.999707 23263 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:05:49.003145 23263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785985105255475-32043-0/minicluster-data/master-1/data/instance:
uuid: "f35d312a79ef46a2b918f8933914bbe0"
format_stamp: "Formatted at 2026-08-06 03:05:48 on dist-test-slave-dhph"
server_key: "005fee1eb26ad9c8abb210a7a0d5e7cd"
server_key_iv: "b1d7490fb7cdbae2eae7b876eb1e60ac"
server_key_version: "encryptionkey@0"
I20260806 03:05:49.004869 23263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785985105255475-32043-0/minicluster-data/master-1/wal/instance:
uuid: "f35d312a79ef46a2b918f8933914bbe0"
format_stamp: "Formatted at 2026-08-06 03:05:48 on dist-test-slave-dhph"
server_key: "005fee1eb26ad9c8abb210a7a0d5e7cd"
server_key_iv: "b1d7490fb7cdbae2eae7b876eb1e60ac"
server_key_version: "encryptionkey@0"
I20260806 03:05:49.013149 23263 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.009s sys 0.000s
I20260806 03:05:49.019279 23279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:05:49.020985 23263 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260806 03:05:49.021320 23263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785985105255475-32043-0/minicluster-data/master-1/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785985105255475-32043-0/minicluster-data/master-1/wal
uuid: "f35d312a79ef46a2b918f8933914bbe0"
format_stamp: "Formatted at 2026-08-06 03:05:48 on dist-test-slave-dhph"
server_key: "005fee1eb26ad9c8abb210a7a0d5e7cd"
server_key_iv: "b1d7490fb7cdbae2eae7b876eb1e60ac"
server_key_version: "encryptionkey@0"
I20260806 03:05:49.021585 23263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785985105255475-32043-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785985105255475-32043-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785985105255475-32043-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:05:49.049465 23263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:05:49.055347 23263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:05:49.055676 23263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:05:49.084233 23263 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.253:38467
I20260806 03:05:49.084281 23331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.253:38467 every 8 connection(s)
I20260806 03:05:49.085656 23263 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785985105255475-32043-0/minicluster-data/master-1/data/info.pb
I20260806 03:05:49.086997 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 23263
I20260806 03:05:49.087210 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785985105255475-32043-0/minicluster-data/master-1/wal/instance
I20260806 03:05:49.087790 32043 external_mini_cluster.cc:1428] Setting key 2a75c4349840f3e281983a8d8affcde7
I20260806 03:05:49.091297 23332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:05:49.105078 23332 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.74.254" port: 37433 } has no permanent_uuid. Determining permanent_uuid...
I20260806 03:05:49.120430 23256 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.74.253" port: 38467 } attempt: 5
Aug 06 03:05:49 dist-test-slave-dhph krb5kdc[23174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985548, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.253@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.74.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.74.252@KRBTEST.COM" created.
Aug 06 03:05:49 dist-test-slave-dhph krb5kdc[23174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985548, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.254@KRBTEST.COM for kudu/127.31.74.253@KRBTEST.COM
I20260806 03:05:49.137459 23335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:49.109250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54714 (local address 127.31.74.254:37433)
0806 03:05:49.109819 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:49.109824 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:49.111100 (+ 1276us) server_negotiation.cc:407] Connection header received
0806 03:05:49.113553 (+ 2453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:49.113560 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:49.113666 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:49.113790 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:49.116495 (+ 2705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:49.117314 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:49.119070 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:49.119506 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:49.119657 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:05:49.128003 (+ 8346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:05:49.128009 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:05:49.128013 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:05:49.128067 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:05:49.132494 (+ 4427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:05:49.133672 (+ 1178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:05:49.133677 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:05:49.133679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:05:49.133862 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:05:49.134481 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:05:49.134486 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:05:49.134488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:05:49.135074 (+ 586us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:05:49.135282 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0806 03:05:49.136401 (+ 1119us) server_negotiation.cc:299] Negotiation successful
0806 03:05:49.136724 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":306,"threads_started":1}
I20260806 03:05:49.138482 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:49.109896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:37433 (local address 127.0.0.1:54714)
0806 03:05:49.110896 (+ 1000us) negotiation.cc:107] Waiting for socket to connect
0806 03:05:49.110949 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 03:05:49.113033 (+ 2084us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:05:49.114347 (+ 1314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:05:49.114358 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:05:49.115180 (+ 822us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:05:49.116173 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:05:49.116235 (+ 62us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:05:49.117507 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:05:49.117512 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:05:49.118901 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:05:49.118915 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:05:49.120068 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:05:49.120086 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:05:49.120438 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:05:49.122001 (+ 1563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:05:49.122054 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:05:49.126180 (+ 4126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:05:49.132658 (+ 6478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:05:49.132668 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:05:49.132690 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:05:49.133434 (+ 744us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:05:49.134052 (+ 618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:05:49.134056 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:05:49.134059 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:05:49.134293 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:05:49.135278 (+ 985us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:05:49.135288 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:05:49.135945 (+ 657us) client_negotiation.cc:769] Sending connection context
0806 03:05:49.136224 (+ 279us) client_negotiation.cc:240] Negotiation successful
0806 03:05:49.136680 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"thread_start_us":405,"threads_started":1}
I20260806 03:05:49.140738 23332 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.74.253" port: 38467 } has no permanent_uuid. Determining permanent_uuid...
I20260806 03:05:49.149331 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:49.121151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.253:38467 (local address 127.0.0.1:57106)
0806 03:05:49.121512 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0806 03:05:49.121533 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 03:05:49.121655 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:05:49.123816 (+ 2161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:05:49.123826 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:05:49.124428 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:05:49.124743 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:05:49.124770 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:05:49.129092 (+ 4322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:05:49.129100 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:05:49.130147 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:05:49.130167 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:05:49.131418 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:05:49.131425 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:05:49.131666 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:05:49.132372 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:05:49.132394 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:05:49.139615 (+ 7221us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:05:49.145898 (+ 6283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:05:49.145906 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:05:49.145910 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:05:49.146606 (+ 696us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:05:49.147229 (+ 623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:05:49.147234 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:05:49.147238 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:05:49.147450 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:05:49.148148 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:05:49.148153 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:05:49.148337 (+ 184us) client_negotiation.cc:769] Sending connection context
0806 03:05:49.148532 (+ 195us) client_negotiation.cc:240] Negotiation successful
0806 03:05:49.148940 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193}
I20260806 03:05:49.150679 23337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:49.121589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57106 (local address 127.31.74.253:38467)
0806 03:05:49.122729 (+ 1140us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:49.122743 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:49.122797 (+ 54us) server_negotiation.cc:407] Connection header received
0806 03:05:49.122885 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:49.122898 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:49.123256 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:49.123585 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:49.125150 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:49.128598 (+ 3448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:49.130339 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:49.131223 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:49.131392 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:05:49.140105 (+ 8713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:05:49.140124 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:05:49.140138 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:05:49.140177 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:05:49.145648 (+ 5471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:05:49.146842 (+ 1194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:05:49.146854 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:05:49.146860 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:05:49.147070 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:05:49.147577 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:05:49.147583 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:05:49.147587 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:05:49.147994 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:05:49.148250 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0806 03:05:49.149770 (+ 1520us) server_negotiation.cc:299] Negotiation successful
0806 03:05:49.150107 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":810,"thread_start_us":590,"threads_started":1}
Aug 06 03:05:49 dist-test-slave-dhph krb5kdc[23174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985548, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.253@KRBTEST.COM for kudu/127.31.74.253@KRBTEST.COM
I20260806 03:05:49.153415 23256 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.74.252" port: 41723 } has no permanent_uuid. Determining permanent_uuid...
I20260806 03:05:49.155656 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:49.154675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.252:41723 (local address 127.0.0.1:47250)
0806 03:05:49.155289 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0806 03:05:49.155427 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.252:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400}
I20260806 03:05:49.157107 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:49.156638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.252:41723 (local address 127.0.0.1:47254)
0806 03:05:49.156839 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0806 03:05:49.156939 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.252:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260806 03:05:49.157477 23256 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.74.252:41723: Network error: Client connection negotiation failed: client connection to 127.31.74.252:41723: connect: Connection refused (error 111)
I20260806 03:05:49.162290 23338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:49.141732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57114 (local address 127.31.74.253:38467)
0806 03:05:49.143050 (+ 1318us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:49.143054 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:49.143075 (+ 21us) server_negotiation.cc:407] Connection header received
0806 03:05:49.143117 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:49.143120 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:49.143176 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:49.143323 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:49.144355 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:49.145560 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:49.147418 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:49.148325 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:49.148470 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:05:49.153814 (+ 5344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:05:49.153820 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:05:49.153825 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:05:49.153864 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:05:49.157407 (+ 3543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:05:49.158357 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:05:49.158366 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:05:49.158369 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:05:49.158511 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:05:49.159984 (+ 1473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:05:49.159992 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:05:49.159995 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:05:49.160585 (+ 590us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:05:49.160842 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0806 03:05:49.161304 (+ 462us) server_negotiation.cc:299] Negotiation successful
0806 03:05:49.161627 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1202,"thread_start_us":995,"threads_started":1}
I20260806 03:05:49.162309 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:49.141907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.253:38467 (local address 127.0.0.1:57114)
0806 03:05:49.142286 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0806 03:05:49.142310 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 03:05:49.142439 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:05:49.143351 (+ 912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:05:49.143357 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:05:49.143957 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:05:49.144121 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:05:49.144130 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:05:49.145698 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:05:49.145704 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:05:49.147254 (+ 1550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:05:49.147270 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:05:49.148495 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:05:49.148504 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:05:49.148703 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:05:49.149424 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:05:49.149452 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:05:49.153577 (+ 4125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:05:49.157619 (+ 4042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:05:49.157628 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:05:49.157634 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:05:49.158223 (+ 589us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:05:49.159037 (+ 814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:05:49.159045 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:05:49.159048 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:05:49.159634 (+ 586us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:05:49.160821 (+ 1187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:05:49.160827 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:05:49.161030 (+ 203us) client_negotiation.cc:769] Sending connection context
0806 03:05:49.161202 (+ 172us) client_negotiation.cc:240] Negotiation successful
0806 03:05:49.161785 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":235}
I20260806 03:05:49.163939 23332 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.74.252" port: 41723 } has no permanent_uuid. Determining permanent_uuid...
I20260806 03:05:49.165555 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:49.164843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.252:41723 (local address 127.0.0.1:47266)
0806 03:05:49.165096 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0806 03:05:49.165314 (+ 218us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.252:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260806 03:05:49.166272 23280 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.74.252:41723: connect: Connection refused (error 111)
I20260806 03:05:49.168684 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:49.167824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.252:41723 (local address 127.0.0.1:47278)
0806 03:05:49.168217 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0806 03:05:49.168411 (+ 194us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.252:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
W20260806 03:05:49.169239 23332 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.74.252:41723: Network error: Client connection negotiation failed: client connection to 127.31.74.252:41723: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.74.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.252.keytab.
Entry for principal kudu/127.31.74.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.252.keytab.
W20260806 03:05:49.196556 32043 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.31.74.252: real 0.108s user 0.006s sys 0.032s
I20260806 03:05:49.197741 32043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
/tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785985105255475-32043-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785985105255475-32043-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785985105255475-32043-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785985105255475-32043-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.252:41723
--webserver_interface=127.31.74.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.252.keytab
--principal=kudu/127.31.74.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.74.212:43523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.74.254:37433,127.31.74.253:38467,127.31.74.252:41723
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785985105255475-32043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785985105255475-32043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785985105255475-32043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 03:05:49.208845 23256 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.74.252" port: 41723 } attempt: 1
I20260806 03:05:49.210322 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:49.209717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.252:41723 (local address 127.0.0.1:47294)
0806 03:05:49.210006 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0806 03:05:49.210118 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.252:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148}
I20260806 03:05:49.212167 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:49.211444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.252:41723 (local address 127.0.0.1:47298)
0806 03:05:49.211927 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0806 03:05:49.212030 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.252:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
W20260806 03:05:49.212611 23256 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.74.252:41723: Network error: Client connection negotiation failed: client connection to 127.31.74.252:41723: connect: Connection refused (error 111)
I20260806 03:05:49.218761 23332 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.74.252" port: 41723 } attempt: 1
I20260806 03:05:49.220199 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:49.219591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.252:41723 (local address 127.0.0.1:47314)
0806 03:05:49.219873 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0806 03:05:49.219976 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.252:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
I20260806 03:05:49.221663 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:49.221023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.252:41723 (local address 127.0.0.1:47326)
0806 03:05:49.221440 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0806 03:05:49.221533 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.252:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254}
W20260806 03:05:49.222498 23332 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.74.252:41723: Network error: Client connection negotiation failed: client connection to 127.31.74.252:41723: connect: Connection refused (error 111)
I20260806 03:05:49.281096 23256 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.74.252" port: 41723 } attempt: 2
I20260806 03:05:49.282948 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:49.282146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.252:41723 (local address 127.0.0.1:47334)
0806 03:05:49.282474 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0806 03:05:49.282670 (+ 196us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.252:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260806 03:05:49.284575 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:49.284020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.252:41723 (local address 127.0.0.1:47350)
0806 03:05:49.284241 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0806 03:05:49.284382 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.252:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260806 03:05:49.284883 23256 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.74.252:41723: Network error: Client connection negotiation failed: client connection to 127.31.74.252:41723: connect: Connection refused (error 111)
I20260806 03:05:49.290877 23332 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.74.252" port: 41723 } attempt: 2
I20260806 03:05:49.292443 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:49.291678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.252:41723 (local address 127.0.0.1:47362)
0806 03:05:49.292135 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0806 03:05:49.292238 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.252:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
I20260806 03:05:49.294188 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:49.293562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.252:41723 (local address 127.0.0.1:47364)
0806 03:05:49.293853 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0806 03:05:49.293987 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.252:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
W20260806 03:05:49.294771 23332 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.74.252:41723: Network error: Client connection negotiation failed: client connection to 127.31.74.252:41723: connect: Connection refused (error 111)
I20260806 03:05:49.386063 23332 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.74.252" port: 41723 } attempt: 3
I20260806 03:05:49.387590 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:49.386972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.252:41723 (local address 127.0.0.1:47376)
0806 03:05:49.387309 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0806 03:05:49.387413 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.252:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184}
I20260806 03:05:49.389235 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:49.388464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.252:41723 (local address 127.0.0.1:47384)
0806 03:05:49.388973 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0806 03:05:49.389090 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.252:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165}
W20260806 03:05:49.389557 23332 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.74.252:41723: Network error: Client connection negotiation failed: client connection to 127.31.74.252:41723: connect: Connection refused (error 111)
I20260806 03:05:49.391160 23256 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.74.252" port: 41723 } attempt: 3
I20260806 03:05:49.393044 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:49.392091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.252:41723 (local address 127.0.0.1:47398)
0806 03:05:49.392621 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0806 03:05:49.392777 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.252:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256}
I20260806 03:05:49.394835 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:49.394140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.252:41723 (local address 127.0.0.1:47412)
0806 03:05:49.394462 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0806 03:05:49.394568 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.252:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163}
W20260806 03:05:49.395128 23256 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.74.252:41723: Network error: Client connection negotiation failed: client connection to 127.31.74.252:41723: connect: Connection refused (error 111)
I20260806 03:05:49.532841 23332 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.74.252" port: 41723 } attempt: 4
I20260806 03:05:49.534435 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:49.533793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.252:41723 (local address 127.0.0.1:47420)
0806 03:05:49.534071 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0806 03:05:49.534168 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.252:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
I20260806 03:05:49.535655 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:49.535283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.252:41723 (local address 127.0.0.1:47436)
0806 03:05:49.535454 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0806 03:05:49.535537 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.252:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260806 03:05:49.535871 23332 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.74.252:41723: Network error: Client connection negotiation failed: client connection to 127.31.74.252:41723: connect: Connection refused (error 111)
I20260806 03:05:49.572417 23256 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.74.252" port: 41723 } attempt: 4
I20260806 03:05:49.573634 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:49.573163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.252:41723 (local address 127.0.0.1:47448)
0806 03:05:49.573377 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0806 03:05:49.573471 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.252:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260806 03:05:49.575098 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:49.574690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.252:41723 (local address 127.0.0.1:47464)
0806 03:05:49.574884 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0806 03:05:49.574965 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.74.252:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260806 03:05:49.575397 23256 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.74.252:41723: Network error: Client connection negotiation failed: client connection to 127.31.74.252:41723: connect: Connection refused (error 111)
W20260806 03:05:49.644136 23341 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 03:05:49.644580 23341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 03:05:49.644713 23341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 03:05:49.644856 23341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 03:05:49.654573 23341 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 03:05:49.654794 23341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 03:05:49.654915 23341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 03:05:49.655004 23341 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 03:05:49.655099 23341 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 03:05:49.666208 23341 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.74.212:43523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785985105255475-32043-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785985105255475-32043-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.31.74.254:37433,127.31.74.253:38467,127.31.74.252:41723
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785985105255475-32043-0/krb5kdc/kudu_127.31.74.252.keytab
--principal=kudu/127.31.74.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.74.252:41723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785985105255475-32043-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.31.74.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785985105255475-32043-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 03:05:49.668048 23341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:05:49.669607 23341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:05:49.682636 23347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:05:49.682564 23346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:05:49.682662 23349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:05:49.683884 23341 server_base.cc:1034] running on GCE node
I20260806 03:05:49.684723 23341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 03:05:49.686349 23341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 03:05:49.687774 23341 hybrid_clock.cc:648] HybridClock initialized: now 1785985549687672 us; error 65 us; skew 500 ppm
I20260806 03:05:49.688243 23341 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 03:05:49 dist-test-slave-dhph krb5kdc[23174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985549, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 03:05:49.691931 23341 init.cc:377] Logged in from keytab as kudu/127.31.74.252@KRBTEST.COM (short username kudu)
I20260806 03:05:49.694181 23341 webserver.cc:460] Webserver started at http://127.31.74.252:42065/ using document root <none> and password file <none>
I20260806 03:05:49.695493 23341 fs_manager.cc:359] Metadata directory not provided
I20260806 03:05:49.695679 23341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:05:49.696033 23341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:05:49.700281 23341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785985105255475-32043-0/minicluster-data/master-2/data/instance:
uuid: "04659e01c08e47b1a75c59eae0fb432b"
format_stamp: "Formatted at 2026-08-06 03:05:49 on dist-test-slave-dhph"
server_key: "80de5faa190d74644b249e0d43f32958"
server_key_iv: "df4cf6acd4ec1533561518df77bb946d"
server_key_version: "encryptionkey@0"
I20260806 03:05:49.701124 23341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785985105255475-32043-0/minicluster-data/master-2/wal/instance:
uuid: "04659e01c08e47b1a75c59eae0fb432b"
format_stamp: "Formatted at 2026-08-06 03:05:49 on dist-test-slave-dhph"
server_key: "80de5faa190d74644b249e0d43f32958"
server_key_iv: "df4cf6acd4ec1533561518df77bb946d"
server_key_version: "encryptionkey@0"
I20260806 03:05:49.707074 23341 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260806 03:05:49.711776 23356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:05:49.713718 23341 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 03:05:49.714104 23341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785985105255475-32043-0/minicluster-data/master-2/data,/tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785985105255475-32043-0/minicluster-data/master-2/wal
uuid: "04659e01c08e47b1a75c59eae0fb432b"
format_stamp: "Formatted at 2026-08-06 03:05:49 on dist-test-slave-dhph"
server_key: "80de5faa190d74644b249e0d43f32958"
server_key_iv: "df4cf6acd4ec1533561518df77bb946d"
server_key_version: "encryptionkey@0"
I20260806 03:05:49.714365 23341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785985105255475-32043-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785985105255475-32043-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785985105255475-32043-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:05:49.740561 23341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:05:49.745698 23341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:05:49.746021 23341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:05:49.771639 23341 rpc_server.cc:307] RPC server started. Bound to: 127.31.74.252:41723
I20260806 03:05:49.771850 23409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.74.252:41723 every 8 connection(s)
I20260806 03:05:49.773233 23341 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785985105255475-32043-0/minicluster-data/master-2/data/info.pb
I20260806 03:05:49.777879 23410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:05:49.780505 32043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu as pid 23341
I20260806 03:05:49.780751 32043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ8w1x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785985105255475-32043-0/minicluster-data/master-2/wal/instance
I20260806 03:05:49.781322 32043 external_mini_cluster.cc:1428] Setting key aaf4758033275e4e610eb42769d90372
I20260806 03:05:49.789371 23410 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.74.254" port: 37433 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 03:05:49 dist-test-slave-dhph krb5kdc[23174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.74.252@KRBTEST.COM
Aug 06 03:05:49 dist-test-slave-dhph krb5kdc[23174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985549, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.252@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
I20260806 03:05:49.810647 23413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:49.783986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47472 (local address 127.31.74.252:41723)
0806 03:05:49.785000 (+ 1014us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:49.785013 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:49.785058 (+ 45us) server_negotiation.cc:407] Connection header received
0806 03:05:49.786914 (+ 1856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:49.786940 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:49.787427 (+ 487us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:49.787880 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:49.789930 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:49.791590 (+ 1660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:49.793517 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:49.794683 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:49.794919 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:05:49.800221 (+ 5302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:05:49.800232 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:05:49.800247 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:05:49.800299 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:05:49.805306 (+ 5007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:05:49.806506 (+ 1200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:05:49.806525 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:05:49.806533 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:05:49.806831 (+ 298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:05:49.807278 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:05:49.807284 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:05:49.807288 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:05:49.807930 (+ 642us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:05:49.808297 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0806 03:05:49.808824 (+ 527us) server_negotiation.cc:299] Negotiation successful
0806 03:05:49.809352 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":721,"thread_start_us":479,"threads_started":1}
I20260806 03:05:49.813567 23416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:49.792289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.254:37433 (local address 127.0.0.1:54722)
0806 03:05:49.793325 (+ 1036us) negotiation.cc:107] Waiting for socket to connect
0806 03:05:49.793373 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 03:05:49.793552 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:05:49.794225 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:05:49.794233 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:05:49.795562 (+ 1329us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:05:49.796173 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:05:49.796191 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:05:49.797429 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:05:49.797434 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:05:49.799480 (+ 2046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:05:49.799498 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:05:49.800826 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:05:49.800834 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:05:49.801626 (+ 792us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:05:49.802556 (+ 930us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:05:49.802663 (+ 107us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:05:49.805971 (+ 3308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:05:49.810042 (+ 4071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:05:49.810056 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:05:49.810064 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:05:49.811184 (+ 1120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:05:49.811648 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:05:49.811653 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:05:49.811657 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:05:49.811810 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:05:49.812388 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:05:49.812396 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:05:49.812601 (+ 205us) client_negotiation.cc:769] Sending connection context
0806 03:05:49.812790 (+ 189us) client_negotiation.cc:240] Negotiation successful
0806 03:05:49.813137 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":759,"thread_start_us":594,"threads_started":1}
I20260806 03:05:49.814157 23414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:49.792290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54722 (local address 127.31.74.254:37433)
0806 03:05:49.792885 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:49.792892 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:49.793548 (+ 656us) server_negotiation.cc:407] Connection header received
0806 03:05:49.793684 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:49.793691 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:49.793784 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:49.793906 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:49.796424 (+ 2518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:49.797232 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:49.799674 (+ 2442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:49.800424 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:49.800833 (+ 409us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:05:49.806250 (+ 5417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:05:49.806256 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:05:49.806260 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:05:49.806312 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:05:49.809866 (+ 3554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:05:49.811370 (+ 1504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:05:49.811374 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:05:49.811376 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:05:49.811518 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:05:49.811937 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:05:49.811941 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:05:49.811943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:05:49.812230 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:05:49.812418 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0806 03:05:49.812895 (+ 477us) server_negotiation.cc:299] Negotiation successful
0806 03:05:49.813140 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":294,"threads_started":1}
I20260806 03:05:49.816447 23410 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.74.253" port: 38467 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 03:05:49 dist-test-slave-dhph krb5kdc[23174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.74.254@KRBTEST.COM
Aug 06 03:05:49 dist-test-slave-dhph krb5kdc[23174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985549, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.252@KRBTEST.COM for kudu/127.31.74.253@KRBTEST.COM
I20260806 03:05:49.835120 23332 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.74.252" port: 41723 } attempt: 5
I20260806 03:05:49.839974 23414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:49.814442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54724 (local address 127.31.74.254:37433)
0806 03:05:49.814668 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:49.814674 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:49.814908 (+ 234us) server_negotiation.cc:407] Connection header received
0806 03:05:49.815218 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:49.815224 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:49.815418 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:49.815546 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:49.817650 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:49.819465 (+ 1815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:49.821564 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:49.822078 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:49.822249 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:05:49.830641 (+ 8392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:05:49.830651 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:05:49.830658 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:05:49.830718 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:05:49.836090 (+ 5372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:05:49.837251 (+ 1161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:05:49.837261 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:05:49.837269 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:05:49.837437 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:05:49.837930 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:05:49.837938 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:05:49.837942 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:05:49.838374 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:05:49.838717 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0806 03:05:49.839291 (+ 574us) server_negotiation.cc:299] Negotiation successful
0806 03:05:49.839565 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":79}
I20260806 03:05:49.841566 23416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:49.818714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.253:38467 (local address 127.0.0.1:57128)
0806 03:05:49.819081 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0806 03:05:49.819103 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 03:05:49.819281 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:05:49.819828 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:05:49.819833 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:05:49.820484 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:05:49.820726 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:05:49.820740 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:05:49.822776 (+ 2036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:05:49.822785 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:05:49.824898 (+ 2113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:05:49.824921 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:05:49.826408 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:05:49.826419 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:05:49.826723 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:05:49.827897 (+ 1174us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:05:49.827919 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:05:49.832118 (+ 4199us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:05:49.837887 (+ 5769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:05:49.837893 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:05:49.837897 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:05:49.838377 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:05:49.838928 (+ 551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:05:49.838934 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:05:49.838939 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:05:49.839140 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:05:49.839960 (+ 820us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:05:49.839967 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:05:49.840152 (+ 185us) client_negotiation.cc:769] Sending connection context
0806 03:05:49.840831 (+ 679us) client_negotiation.cc:240] Negotiation successful
0806 03:05:49.841169 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225}
I20260806 03:05:49.841723 23417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:49.818490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57128 (local address 127.31.74.253:38467)
0806 03:05:49.819091 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:49.819099 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:49.819190 (+ 91us) server_negotiation.cc:407] Connection header received
0806 03:05:49.819444 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:49.819449 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:49.819550 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:49.819890 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:49.821153 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:49.822499 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:49.825740 (+ 3241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:49.826175 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:49.826294 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:05:49.832599 (+ 6305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:05:49.832606 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:05:49.832647 (+ 41us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:05:49.832721 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:05:49.837679 (+ 4958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:05:49.838555 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:05:49.838559 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:05:49.838562 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:05:49.838788 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:05:49.839301 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:05:49.839306 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:05:49.839310 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:05:49.839696 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:05:49.839875 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0806 03:05:49.841038 (+ 1163us) server_negotiation.cc:299] Negotiation successful
0806 03:05:49.841337 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":327,"threads_started":1}
I20260806 03:05:49.843252 23410 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.74.252" port: 41723 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 03:05:49 dist-test-slave-dhph krb5kdc[23174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985548, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.253@KRBTEST.COM for kudu/127.31.74.252@KRBTEST.COM
I20260806 03:05:49.852636 23256 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.74.252" port: 41723 } attempt: 5
Aug 06 03:05:49 dist-test-slave-dhph krb5kdc[23174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985549, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.252@KRBTEST.COM for kudu/127.31.74.252@KRBTEST.COM
I20260806 03:05:49.857035 23413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:49.835986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47482 (local address 127.31.74.252:41723)
0806 03:05:49.836200 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:49.836208 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:49.836289 (+ 81us) server_negotiation.cc:407] Connection header received
0806 03:05:49.836521 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:49.836528 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:49.836605 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:49.836721 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:49.837800 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:49.839527 (+ 1727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:49.841389 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:49.842080 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:49.842229 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:05:49.848603 (+ 6374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:05:49.848610 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:05:49.848615 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:05:49.848651 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:05:49.852587 (+ 3936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:05:49.853528 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:05:49.853533 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:05:49.853536 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:05:49.853701 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:05:49.854189 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:05:49.854195 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:05:49.854203 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:05:49.855152 (+ 949us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:05:49.855365 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0806 03:05:49.856243 (+ 878us) server_negotiation.cc:299] Negotiation successful
0806 03:05:49.856523 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
Aug 06 03:05:49 dist-test-slave-dhph krb5kdc[23174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.74.253@KRBTEST.COM
I20260806 03:05:49.858507 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:49.835877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.252:41723 (local address 127.0.0.1:47482)
0806 03:05:49.836197 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0806 03:05:49.836220 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 03:05:49.836350 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:05:49.836746 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:05:49.836752 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:05:49.837362 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:05:49.837650 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:05:49.837662 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:05:49.839690 (+ 2028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:05:49.839696 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:05:49.841210 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:05:49.841227 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:05:49.842415 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:05:49.842427 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:05:49.842685 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:05:49.843617 (+ 932us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:05:49.843657 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:05:49.848393 (+ 4736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:05:49.852781 (+ 4388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:05:49.852787 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:05:49.852790 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:05:49.853373 (+ 583us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:05:49.853846 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:05:49.853851 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:05:49.853855 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:05:49.854029 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:05:49.855323 (+ 1294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:05:49.855330 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:05:49.855529 (+ 199us) client_negotiation.cc:769] Sending connection context
0806 03:05:49.857565 (+ 2036us) client_negotiation.cc:240] Negotiation successful
0806 03:05:49.858009 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179}
I20260806 03:05:49.866125 23416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:49.844307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.252:41723 (local address 127.0.0.1:47492)
0806 03:05:49.844522 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0806 03:05:49.844545 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 03:05:49.844661 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:05:49.846860 (+ 2199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:05:49.846865 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:05:49.847329 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:05:49.847459 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:05:49.847467 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:05:49.849056 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:05:49.849063 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:05:49.850741 (+ 1678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:05:49.850758 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:05:49.852047 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:05:49.852053 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:05:49.852253 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:05:49.852814 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:05:49.852863 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:05:49.856414 (+ 3551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:05:49.862526 (+ 6112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:05:49.862534 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:05:49.862539 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:05:49.863654 (+ 1115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:05:49.864168 (+ 514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:05:49.864174 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:05:49.864177 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:05:49.864440 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:05:49.865094 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:05:49.865100 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:05:49.865268 (+ 168us) client_negotiation.cc:769] Sending connection context
0806 03:05:49.865398 (+ 130us) client_negotiation.cc:240] Negotiation successful
0806 03:05:49.865741 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
Aug 06 03:05:49 dist-test-slave-dhph krb5kdc[23174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985548, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.74.254@KRBTEST.COM for kudu/127.31.74.252@KRBTEST.COM
I20260806 03:05:49.867230 23418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:49.844567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47492 (local address 127.31.74.252:41723)
0806 03:05:49.846514 (+ 1947us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:49.846522 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:49.846551 (+ 29us) server_negotiation.cc:407] Connection header received
0806 03:05:49.846629 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:49.846634 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:49.846718 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:49.846841 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:49.847616 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:49.848883 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:49.850919 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:49.851861 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:49.852026 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:05:49.856677 (+ 4651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:05:49.856685 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:05:49.856697 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:05:49.856734 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:05:49.862216 (+ 5482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:05:49.863840 (+ 1624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:05:49.863845 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:05:49.863849 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:05:49.864031 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:05:49.864592 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:05:49.864598 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:05:49.864602 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:05:49.864960 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:05:49.866423 (+ 1463us) server_negotiation.cc:1035] Waiting for connection context
0806 03:05:49.866605 (+ 182us) server_negotiation.cc:299] Negotiation successful
0806 03:05:49.866852 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1684,"thread_start_us":1461,"threads_started":1}
I20260806 03:05:49.869200 23332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f35d312a79ef46a2b918f8933914bbe0: Bootstrap starting.
I20260806 03:05:49.873397 23332 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f35d312a79ef46a2b918f8933914bbe0: Neither blocks nor log segments found. Creating new log.
I20260806 03:05:49.873891 23417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:49.842275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57130 (local address 127.31.74.253:38467)
0806 03:05:49.842736 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:49.842741 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:49.842755 (+ 14us) server_negotiation.cc:407] Connection header received
0806 03:05:49.842794 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:49.842799 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:49.842937 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:49.843035 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:49.846112 (+ 3077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:49.847726 (+ 1614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:49.849666 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:49.850427 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:49.850568 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:05:49.861168 (+ 10600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:05:49.861177 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:05:49.861184 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:05:49.861234 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:05:49.865966 (+ 4732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:05:49.867598 (+ 1632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:05:49.867606 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:05:49.867609 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:05:49.867850 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:05:49.870800 (+ 2950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:05:49.870813 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:05:49.870818 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:05:49.871419 (+ 601us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:05:49.871725 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0806 03:05:49.873283 (+ 1558us) server_negotiation.cc:299] Negotiation successful
0806 03:05:49.873556 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328}
I20260806 03:05:49.875119 23332 log.cc:824] T 00000000000000000000000000000000 P f35d312a79ef46a2b918f8933914bbe0: Log is configured to *not* fsync() on all Append() calls
I20260806 03:05:49.875226 23410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04659e01c08e47b1a75c59eae0fb432b: Bootstrap starting.
I20260806 03:05:49.878959 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:49.853330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.74.252:41723 (local address 127.0.0.1:47508)
0806 03:05:49.853542 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0806 03:05:49.853564 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 03:05:49.853673 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 03:05:49.856544 (+ 2871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 03:05:49.856552 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 03:05:49.857140 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0806 03:05:49.857395 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:05:49.857410 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:05:49.859785 (+ 2375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:05:49.859793 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:05:49.861461 (+ 1668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 03:05:49.861480 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 03:05:49.862894 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 03:05:49.862903 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 03:05:49.863126 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:05:49.864223 (+ 1097us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 03:05:49.864251 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 03:05:49.867986 (+ 3735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 03:05:49.873459 (+ 5473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:05:49.873466 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:05:49.873472 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:05:49.874170 (+ 698us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:05:49.876528 (+ 2358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 03:05:49.876534 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 03:05:49.876539 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 03:05:49.876814 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 03:05:49.877693 (+ 879us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 03:05:49.877700 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 03:05:49.877946 (+ 246us) client_negotiation.cc:769] Sending connection context
0806 03:05:49.878115 (+ 169us) client_negotiation.cc:240] Negotiation successful
0806 03:05:49.878458 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260806 03:05:49.879655 23332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f35d312a79ef46a2b918f8933914bbe0: No bootstrap required, opened a new log
I20260806 03:05:49.879930 23410 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04659e01c08e47b1a75c59eae0fb432b: Neither blocks nor log segments found. Creating new log.
I20260806 03:05:49.880131 32043 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 03:05:49.880890 23419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:49.853876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47508 (local address 127.31.74.252:41723)
0806 03:05:49.856075 (+ 2199us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:49.856083 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:49.856118 (+ 35us) server_negotiation.cc:407] Connection header received
0806 03:05:49.856175 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:49.856182 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:49.856272 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:49.857575 (+ 1303us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:49.857887 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:49.859579 (+ 1692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:49.861707 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:49.862672 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:49.868256 (+ 5584us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:05:49.868719 (+ 463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:05:49.868735 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:05:49.868741 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:05:49.868848 (+ 107us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:05:49.873212 (+ 4364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:05:49.874398 (+ 1186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:05:49.874403 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:05:49.874407 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:05:49.875337 (+ 930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:05:49.877008 (+ 1671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:05:49.877015 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:05:49.877019 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:05:49.877512 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:05:49.879998 (+ 2486us) server_negotiation.cc:1035] Waiting for connection context
0806 03:05:49.880151 (+ 153us) server_negotiation.cc:299] Negotiation successful
0806 03:05:49.880445 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1982,"thread_start_us":1528,"threads_started":1}
I20260806 03:05:49.883003 23410 log.cc:824] T 00000000000000000000000000000000 P 04659e01c08e47b1a75c59eae0fb432b: Log is configured to *not* fsync() on all Append() calls
I20260806 03:05:49.887625 23256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e99b8e574f924612ab48cc2a396a169a: Bootstrap starting.
I20260806 03:05:49.884219 23229 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "e99b8e574f924612ab48cc2a396a169a"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54724
I20260806 03:05:49.892963 23332 raft_consensus.cc:348] T 00000000000000000000000000000000 P f35d312a79ef46a2b918f8933914bbe0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99b8e574f924612ab48cc2a396a169a" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 37433 } } peers { permanent_uuid: "f35d312a79ef46a2b918f8933914bbe0" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 38467 } } peers { permanent_uuid: "04659e01c08e47b1a75c59eae0fb432b" member_type: VOTER last_known_addr { host: "127.31.74.252" port: 41723 } }
I20260806 03:05:49.893515 23332 raft_consensus.cc:729] T 00000000000000000000000000000000 P f35d312a79ef46a2b918f8933914bbe0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f35d312a79ef46a2b918f8933914bbe0, State: Initialized, Role: FOLLOWER
I20260806 03:05:49.893674 23256 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e99b8e574f924612ab48cc2a396a169a: Neither blocks nor log segments found. Creating new log.
I20260806 03:05:49.894192 23332 consensus_queue.cc:260] T 00000000000000000000000000000000 P f35d312a79ef46a2b918f8933914bbe0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99b8e574f924612ab48cc2a396a169a" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 37433 } } peers { permanent_uuid: "f35d312a79ef46a2b918f8933914bbe0" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 38467 } } peers { permanent_uuid: "04659e01c08e47b1a75c59eae0fb432b" member_type: VOTER last_known_addr { host: "127.31.74.252" port: 41723 } }
I20260806 03:05:49.894456 23410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04659e01c08e47b1a75c59eae0fb432b: No bootstrap required, opened a new log
I20260806 03:05:49.895145 23256 log.cc:824] T 00000000000000000000000000000000 P e99b8e574f924612ab48cc2a396a169a: Log is configured to *not* fsync() on all Append() calls
I20260806 03:05:49.895816 23422 sys_catalog.cc:455] T 00000000000000000000000000000000 P f35d312a79ef46a2b918f8933914bbe0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99b8e574f924612ab48cc2a396a169a" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 37433 } } peers { permanent_uuid: "f35d312a79ef46a2b918f8933914bbe0" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 38467 } } peers { permanent_uuid: "04659e01c08e47b1a75c59eae0fb432b" member_type: VOTER last_known_addr { host: "127.31.74.252" port: 41723 } } }
I20260806 03:05:49.896921 23332 sys_catalog.cc:564] T 00000000000000000000000000000000 P f35d312a79ef46a2b918f8933914bbe0 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:05:49.897149 23422 sys_catalog.cc:458] T 00000000000000000000000000000000 P f35d312a79ef46a2b918f8933914bbe0 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 03:05:49.906864 23256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e99b8e574f924612ab48cc2a396a169a: No bootstrap required, opened a new log
I20260806 03:05:49.910032 23410 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04659e01c08e47b1a75c59eae0fb432b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99b8e574f924612ab48cc2a396a169a" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 37433 } } peers { permanent_uuid: "f35d312a79ef46a2b918f8933914bbe0" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 38467 } } peers { permanent_uuid: "04659e01c08e47b1a75c59eae0fb432b" member_type: VOTER last_known_addr { host: "127.31.74.252" port: 41723 } }
I20260806 03:05:49.910727 23410 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04659e01c08e47b1a75c59eae0fb432b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04659e01c08e47b1a75c59eae0fb432b, State: Initialized, Role: FOLLOWER
I20260806 03:05:49.911441 23410 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04659e01c08e47b1a75c59eae0fb432b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99b8e574f924612ab48cc2a396a169a" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 37433 } } peers { permanent_uuid: "f35d312a79ef46a2b918f8933914bbe0" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 38467 } } peers { permanent_uuid: "04659e01c08e47b1a75c59eae0fb432b" member_type: VOTER last_known_addr { host: "127.31.74.252" port: 41723 } }
I20260806 03:05:49.918823 23256 raft_consensus.cc:348] T 00000000000000000000000000000000 P e99b8e574f924612ab48cc2a396a169a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99b8e574f924612ab48cc2a396a169a" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 37433 } } peers { permanent_uuid: "f35d312a79ef46a2b918f8933914bbe0" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 38467 } } peers { permanent_uuid: "04659e01c08e47b1a75c59eae0fb432b" member_type: VOTER last_known_addr { host: "127.31.74.252" port: 41723 } }
I20260806 03:05:49.919349 23256 raft_consensus.cc:729] T 00000000000000000000000000000000 P e99b8e574f924612ab48cc2a396a169a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e99b8e574f924612ab48cc2a396a169a, State: Initialized, Role: FOLLOWER
I20260806 03:05:49.920018 23256 consensus_queue.cc:260] T 00000000000000000000000000000000 P e99b8e574f924612ab48cc2a396a169a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99b8e574f924612ab48cc2a396a169a" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 37433 } } peers { permanent_uuid: "f35d312a79ef46a2b918f8933914bbe0" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 38467 } } peers { permanent_uuid: "04659e01c08e47b1a75c59eae0fb432b" member_type: VOTER last_known_addr { host: "127.31.74.252" port: 41723 } }
I20260806 03:05:49.921525 23432 sys_catalog.cc:455] T 00000000000000000000000000000000 P e99b8e574f924612ab48cc2a396a169a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99b8e574f924612ab48cc2a396a169a" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 37433 } } peers { permanent_uuid: "f35d312a79ef46a2b918f8933914bbe0" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 38467 } } peers { permanent_uuid: "04659e01c08e47b1a75c59eae0fb432b" member_type: VOTER last_known_addr { host: "127.31.74.252" port: 41723 } } }
I20260806 03:05:49.922330 23432 sys_catalog.cc:458] T 00000000000000000000000000000000 P e99b8e574f924612ab48cc2a396a169a [sys.catalog]: This master's current role is: FOLLOWER
I20260806 03:05:49.921823 23431 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04659e01c08e47b1a75c59eae0fb432b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99b8e574f924612ab48cc2a396a169a" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 37433 } } peers { permanent_uuid: "f35d312a79ef46a2b918f8933914bbe0" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 38467 } } peers { permanent_uuid: "04659e01c08e47b1a75c59eae0fb432b" member_type: VOTER last_known_addr { host: "127.31.74.252" port: 41723 } } }
I20260806 03:05:49.923214 23431 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04659e01c08e47b1a75c59eae0fb432b [sys.catalog]: This master's current role is: FOLLOWER
I20260806 03:05:49.925490 23410 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04659e01c08e47b1a75c59eae0fb432b [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:05:49.925998 23256 sys_catalog.cc:564] T 00000000000000000000000000000000 P e99b8e574f924612ab48cc2a396a169a [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:05:49.926086 23229 raft_consensus.cc:482] T 00000000000000000000000000000000 P e99b8e574f924612ab48cc2a396a169a [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:05:49.926367 23229 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e99b8e574f924612ab48cc2a396a169a [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:05:49.931990 23229 raft_consensus.cc:504] T 00000000000000000000000000000000 P e99b8e574f924612ab48cc2a396a169a [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99b8e574f924612ab48cc2a396a169a" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 37433 } } peers { permanent_uuid: "f35d312a79ef46a2b918f8933914bbe0" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 38467 } } peers { permanent_uuid: "04659e01c08e47b1a75c59eae0fb432b" member_type: VOTER last_known_addr { host: "127.31.74.252" port: 41723 } }
I20260806 03:05:49.935736 23384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e99b8e574f924612ab48cc2a396a169a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "04659e01c08e47b1a75c59eae0fb432b"
I20260806 03:05:49.935736 23306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e99b8e574f924612ab48cc2a396a169a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f35d312a79ef46a2b918f8933914bbe0"
I20260806 03:05:49.936371 23384 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04659e01c08e47b1a75c59eae0fb432b [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:05:49.936373 23306 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f35d312a79ef46a2b918f8933914bbe0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:05:49.940919 23384 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 04659e01c08e47b1a75c59eae0fb432b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e99b8e574f924612ab48cc2a396a169a in term 1.
I20260806 03:05:49.941032 23306 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f35d312a79ef46a2b918f8933914bbe0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e99b8e574f924612ab48cc2a396a169a in term 1.
I20260806 03:05:49.941835 23205 leader_election.cc:302] T 00000000000000000000000000000000 P e99b8e574f924612ab48cc2a396a169a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e99b8e574f924612ab48cc2a396a169a, f35d312a79ef46a2b918f8933914bbe0; no voters:
I20260806 03:05:49.942173 23229 leader_election.cc:288] T 00000000000000000000000000000000 P e99b8e574f924612ab48cc2a396a169a [CANDIDATE]: Term 1 election: Requested vote from peers f35d312a79ef46a2b918f8933914bbe0 (127.31.74.253:38467), 04659e01c08e47b1a75c59eae0fb432b (127.31.74.252:41723)
I20260806 03:05:49.942487 23432 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e99b8e574f924612ab48cc2a396a169a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:05:49.968904 23432 raft_consensus.cc:686] T 00000000000000000000000000000000 P e99b8e574f924612ab48cc2a396a169a [term 1 LEADER]: Becoming Leader. State: Replica: e99b8e574f924612ab48cc2a396a169a, State: Running, Role: LEADER
I20260806 03:05:49.970070 23432 consensus_queue.cc:237] T 00000000000000000000000000000000 P e99b8e574f924612ab48cc2a396a169a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99b8e574f924612ab48cc2a396a169a" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 37433 } } peers { permanent_uuid: "f35d312a79ef46a2b918f8933914bbe0" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 38467 } } peers { permanent_uuid: "04659e01c08e47b1a75c59eae0fb432b" member_type: VOTER last_known_addr { host: "127.31.74.252" port: 41723 } }
I20260806 03:05:49.981485 23438 sys_catalog.cc:455] T 00000000000000000000000000000000 P e99b8e574f924612ab48cc2a396a169a [sys.catalog]: SysCatalogTable state changed. Reason: New leader e99b8e574f924612ab48cc2a396a169a. Latest consensus state: current_term: 1 leader_uuid: "e99b8e574f924612ab48cc2a396a169a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99b8e574f924612ab48cc2a396a169a" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 37433 } } peers { permanent_uuid: "f35d312a79ef46a2b918f8933914bbe0" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 38467 } } peers { permanent_uuid: "04659e01c08e47b1a75c59eae0fb432b" member_type: VOTER last_known_addr { host: "127.31.74.252" port: 41723 } } }
I20260806 03:05:49.982237 23438 sys_catalog.cc:458] T 00000000000000000000000000000000 P e99b8e574f924612ab48cc2a396a169a [sys.catalog]: This master's current role is: LEADER
I20260806 03:05:49.991578 23447 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260806 03:05:49.998883 23362 tablet.cc:2369] T 00000000000000000000000000000000 P 04659e01c08e47b1a75c59eae0fb432b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 03:05:50.027911 23452 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 03:05:50.031428 23263 master_runner.cc:186] Error getting master registration for 127.31.74.254:37433: OK, instance_id { permanent_uuid: "e99b8e574f924612ab48cc2a396a169a" instance_seqno: 1785985548395314 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 03:05:50.049192 23263 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260806 03:05:50.058962 23285 tablet.cc:2369] T 00000000000000000000000000000000 P f35d312a79ef46a2b918f8933914bbe0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 03:05:50.063783 23447 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:05:50.054457 23341 master_runner.cc:186] Error getting master registration for 127.31.74.254:37433: OK, instance_id { permanent_uuid: "e99b8e574f924612ab48cc2a396a169a" instance_seqno: 1785985548395314 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 03:05:50.068110 23341 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 03:05:50.072532 23419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:49.984977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47510 (local address 127.31.74.252:41723)
0806 03:05:49.985279 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:49.985290 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:49.998738 (+ 13448us) server_negotiation.cc:407] Connection header received
0806 03:05:49.999082 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:49.999088 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:49.999220 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:49.999304 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:50.019568 (+ 20264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:50.020712 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:50.029192 (+ 8480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:50.030393 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:50.030566 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:05:50.050242 (+ 19676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:05:50.050260 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:05:50.050269 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:05:50.050392 (+ 123us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:05:50.056855 (+ 6463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:05:50.069822 (+ 12967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:05:50.069829 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:05:50.069833 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:05:50.070020 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:05:50.070520 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:05:50.070526 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:05:50.070529 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:05:50.070957 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:05:50.071202 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0806 03:05:50.071890 (+ 688us) server_negotiation.cc:299] Negotiation successful
0806 03:05:50.072162 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260806 03:05:50.073189 23462 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 04659e01c08e47b1a75c59eae0fb432b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 03:05:50.073312 23462 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 03:05:50.076751 23414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:49.986305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54730 (local address 127.31.74.254:37433)
0806 03:05:49.986574 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:49.986609 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:50.002886 (+ 16277us) server_negotiation.cc:407] Connection header received
0806 03:05:50.003218 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:50.003227 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:50.003352 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:50.003447 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:50.020224 (+ 16777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:50.022698 (+ 2474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:50.034951 (+ 12253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:50.035952 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:50.036086 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:05:50.064507 (+ 28421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:05:50.064515 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:05:50.064522 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:05:50.064574 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:05:50.068986 (+ 4412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:05:50.073136 (+ 4150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:05:50.073142 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:05:50.073145 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:05:50.073632 (+ 487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:05:50.074167 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:05:50.074172 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:05:50.074176 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:05:50.074644 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:05:50.075056 (+ 412us) server_negotiation.cc:1035] Waiting for connection context
0806 03:05:50.075966 (+ 910us) server_negotiation.cc:299] Negotiation successful
0806 03:05:50.076319 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 03:05:50.080476 23384 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 04659e01c08e47b1a75c59eae0fb432b [term 1 FOLLOWER]: Refusing update from remote peer e99b8e574f924612ab48cc2a396a169a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:05:50.081452 23306 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f35d312a79ef46a2b918f8933914bbe0 [term 1 FOLLOWER]: Refusing update from remote peer e99b8e574f924612ab48cc2a396a169a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:05:50.082108 23432 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e99b8e574f924612ab48cc2a396a169a [LEADER]: Connected to new peer: Peer: permanent_uuid: "04659e01c08e47b1a75c59eae0fb432b" member_type: VOTER last_known_addr { host: "127.31.74.252" port: 41723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:05:50.082489 23438 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e99b8e574f924612ab48cc2a396a169a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f35d312a79ef46a2b918f8933914bbe0" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 38467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:05:50.110898 23417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:49.989147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57144 (local address 127.31.74.253:38467)
0806 03:05:49.989432 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:49.989442 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:50.007671 (+ 18229us) server_negotiation.cc:407] Connection header received
0806 03:05:50.008230 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:50.008242 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:50.008391 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:50.008487 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:50.015391 (+ 6904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:50.017478 (+ 2087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:50.044396 (+ 26918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:50.045752 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:50.045968 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:05:50.078217 (+ 32249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 03:05:50.078225 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 03:05:50.078230 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 03:05:50.078334 (+ 104us) server_negotiation.cc:922] calling sasl_server_start()
0806 03:05:50.099851 (+ 21517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:05:50.107504 (+ 7653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:05:50.107512 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:05:50.107516 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:05:50.107760 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 03:05:50.108354 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 03:05:50.108360 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 03:05:50.108364 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 03:05:50.109100 (+ 736us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 03:05:50.109429 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0806 03:05:50.110106 (+ 677us) server_negotiation.cc:299] Negotiation successful
0806 03:05:50.110418 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260806 03:05:50.134279 23208 tablet.cc:2369] T 00000000000000000000000000000000 P e99b8e574f924612ab48cc2a396a169a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 03:05:50.137239 23431 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04659e01c08e47b1a75c59eae0fb432b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e99b8e574f924612ab48cc2a396a169a. Latest consensus state: current_term: 1 leader_uuid: "e99b8e574f924612ab48cc2a396a169a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99b8e574f924612ab48cc2a396a169a" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 37433 } } peers { permanent_uuid: "f35d312a79ef46a2b918f8933914bbe0" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 38467 } } peers { permanent_uuid: "04659e01c08e47b1a75c59eae0fb432b" member_type: VOTER last_known_addr { host: "127.31.74.252" port: 41723 } } }
I20260806 03:05:50.137605 23431 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04659e01c08e47b1a75c59eae0fb432b [sys.catalog]: This master's current role is: FOLLOWER
I20260806 03:05:50.138715 23422 sys_catalog.cc:455] T 00000000000000000000000000000000 P f35d312a79ef46a2b918f8933914bbe0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e99b8e574f924612ab48cc2a396a169a. Latest consensus state: current_term: 1 leader_uuid: "e99b8e574f924612ab48cc2a396a169a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99b8e574f924612ab48cc2a396a169a" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 37433 } } peers { permanent_uuid: "f35d312a79ef46a2b918f8933914bbe0" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 38467 } } peers { permanent_uuid: "04659e01c08e47b1a75c59eae0fb432b" member_type: VOTER last_known_addr { host: "127.31.74.252" port: 41723 } } }
I20260806 03:05:50.139046 23422 sys_catalog.cc:458] T 00000000000000000000000000000000 P f35d312a79ef46a2b918f8933914bbe0 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 03:05:50.140789 23465 mvcc.cc:205] Tried to move back new op lower bound from 7315396813116973056 to 7315396812687228928. Current Snapshot: MvccSnapshot[applied={T|T < 7315396813116973056}]
I20260806 03:05:50.143430 23468 mvcc.cc:205] Tried to move back new op lower bound from 7315396813116973056 to 7315396812687228928. Current Snapshot: MvccSnapshot[applied={T|T < 7315396813116973056}]
I20260806 03:05:50.143565 23431 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04659e01c08e47b1a75c59eae0fb432b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e99b8e574f924612ab48cc2a396a169a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99b8e574f924612ab48cc2a396a169a" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 37433 } } peers { permanent_uuid: "f35d312a79ef46a2b918f8933914bbe0" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 38467 } } peers { permanent_uuid: "04659e01c08e47b1a75c59eae0fb432b" member_type: VOTER last_known_addr { host: "127.31.74.252" port: 41723 } } }
I20260806 03:05:50.143810 23431 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04659e01c08e47b1a75c59eae0fb432b [sys.catalog]: This master's current role is: FOLLOWER
I20260806 03:05:50.146910 23422 sys_catalog.cc:455] T 00000000000000000000000000000000 P f35d312a79ef46a2b918f8933914bbe0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e99b8e574f924612ab48cc2a396a169a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99b8e574f924612ab48cc2a396a169a" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 37433 } } peers { permanent_uuid: "f35d312a79ef46a2b918f8933914bbe0" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 38467 } } peers { permanent_uuid: "04659e01c08e47b1a75c59eae0fb432b" member_type: VOTER last_known_addr { host: "127.31.74.252" port: 41723 } } }
I20260806 03:05:50.147230 23422 sys_catalog.cc:458] T 00000000000000000000000000000000 P f35d312a79ef46a2b918f8933914bbe0 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 03:05:50.150126 23467 mvcc.cc:205] Tried to move back new op lower bound from 7315396813116973056 to 7315396812687228928. Current Snapshot: MvccSnapshot[applied={T|T < 7315396813116973056}]
I20260806 03:05:50.166463 23438 sys_catalog.cc:455] T 00000000000000000000000000000000 P e99b8e574f924612ab48cc2a396a169a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e99b8e574f924612ab48cc2a396a169a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99b8e574f924612ab48cc2a396a169a" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 37433 } } peers { permanent_uuid: "f35d312a79ef46a2b918f8933914bbe0" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 38467 } } peers { permanent_uuid: "04659e01c08e47b1a75c59eae0fb432b" member_type: VOTER last_known_addr { host: "127.31.74.252" port: 41723 } } }
I20260806 03:05:50.166934 23438 sys_catalog.cc:458] T 00000000000000000000000000000000 P e99b8e574f924612ab48cc2a396a169a [sys.catalog]: This master's current role is: LEADER
I20260806 03:05:50.167272 23438 sys_catalog.cc:455] T 00000000000000000000000000000000 P e99b8e574f924612ab48cc2a396a169a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e99b8e574f924612ab48cc2a396a169a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99b8e574f924612ab48cc2a396a169a" member_type: VOTER last_known_addr { host: "127.31.74.254" port: 37433 } } peers { permanent_uuid: "f35d312a79ef46a2b918f8933914bbe0" member_type: VOTER last_known_addr { host: "127.31.74.253" port: 38467 } } peers { permanent_uuid: "04659e01c08e47b1a75c59eae0fb432b" member_type: VOTER last_known_addr { host: "127.31.74.252" port: 41723 } } }
I20260806 03:05:50.167613 23438 sys_catalog.cc:458] T 00000000000000000000000000000000 P e99b8e574f924612ab48cc2a396a169a [sys.catalog]: This master's current role is: LEADER
I20260806 03:05:50.170990 23447 catalog_manager.cc:1347] Generated new cluster ID: d33a3bdbfbcd43f1b19cc835e64dafb7
I20260806 03:05:50.171115 23447 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:05:50.219050 23447 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:05:50.221848 23447 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:05:50.238894 23479 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 03:05:50.261893 23447 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e99b8e574f924612ab48cc2a396a169a: Generated new TSK 0
I20260806 03:05:50.263260 23447 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 03:05:50.396849 23417 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:05:50.382412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57158 (local address 127.31.74.253:38467)
0806 03:05:50.382611 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:50.382618 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:50.393653 (+ 11035us) server_negotiation.cc:407] Connection header received
0806 03:05:50.394376 (+ 723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:50.394386 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:50.394519 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:50.394641 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:50.396363 (+ 1722us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57158: BlockingRecv error: recv got EOF from 127.0.0.1:57158 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260806 03:05:50.400508 23414 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:05:50.382261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54746 (local address 127.31.74.254:37433)
0806 03:05:50.382490 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:50.382498 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:50.397983 (+ 15485us) server_negotiation.cc:407] Connection header received
0806 03:05:50.398367 (+ 384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:50.398376 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:50.398495 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:50.398601 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:50.399886 (+ 1285us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54746: BlockingRecv error: recv got EOF from 127.0.0.1:54746 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260806 03:05:50.403700 23419 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:05:50.383708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47520 (local address 127.31.74.252:41723)
0806 03:05:50.383927 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:50.383934 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:50.401546 (+ 17612us) server_negotiation.cc:407] Connection header received
0806 03:05:50.401906 (+ 360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:50.401915 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:50.402025 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:50.402117 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:50.403339 (+ 1222us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47520: BlockingRecv error: recv got EOF from 127.0.0.1:47520 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260806 03:05:50.437032 23417 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:05:50.431073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57160 (local address 127.31.74.253:38467)
0806 03:05:50.431283 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:50.431290 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:50.435254 (+ 3964us) server_negotiation.cc:407] Connection header received
0806 03:05:50.435473 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:50.435481 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:50.435622 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:50.435751 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:50.436786 (+ 1035us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57160: BlockingRecv error: recv got EOF from 127.0.0.1:57160 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260806 03:05:50.444823 23417 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:05:50.441905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57172 (local address 127.31.74.253:38467)
0806 03:05:50.442091 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:50.442098 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:50.443207 (+ 1109us) server_negotiation.cc:407] Connection header received
0806 03:05:50.443432 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:50.443446 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:50.443535 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:50.443653 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:50.444561 (+ 908us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57172: BlockingRecv error: recv got EOF from 127.0.0.1:57172 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260806 03:05:50.455088 23417 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:05:50.452696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57188 (local address 127.31.74.253:38467)
0806 03:05:50.452884 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:50.452891 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:50.453372 (+ 481us) server_negotiation.cc:407] Connection header received
0806 03:05:50.453587 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:50.453595 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:50.453725 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:50.453857 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:50.454810 (+ 953us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57188: BlockingRecv error: recv got EOF from 127.0.0.1:57188 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260806 03:05:50.465313 23417 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:05:50.462937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57194 (local address 127.31.74.253:38467)
0806 03:05:50.463127 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:50.463133 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:50.463681 (+ 548us) server_negotiation.cc:407] Connection header received
0806 03:05:50.463879 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:50.463886 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:50.464008 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:50.464138 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:50.465048 (+ 910us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57194: BlockingRecv error: recv got EOF from 127.0.0.1:57194 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260806 03:05:50.479852 23417 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:05:50.476869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57208 (local address 127.31.74.253:38467)
0806 03:05:50.477047 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:50.477053 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:50.477663 (+ 610us) server_negotiation.cc:407] Connection header received
0806 03:05:50.478361 (+ 698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:50.478367 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:50.478483 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:50.478690 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:50.479550 (+ 860us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57208: BlockingRecv error: recv got EOF from 127.0.0.1:57208 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260806 03:05:50.502822 23417 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:05:50.499954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57224 (local address 127.31.74.253:38467)
0806 03:05:50.500177 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:50.500184 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:50.500734 (+ 550us) server_negotiation.cc:407] Connection header received
0806 03:05:50.501385 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:50.501391 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:50.501515 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:50.501646 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:50.502554 (+ 908us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57224: BlockingRecv error: recv got EOF from 127.0.0.1:57224 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260806 03:05:50.547328 23417 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:05:50.544799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57240 (local address 127.31.74.253:38467)
0806 03:05:50.545060 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:50.545066 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:50.545604 (+ 538us) server_negotiation.cc:407] Connection header received
0806 03:05:50.545847 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:50.545853 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:50.545963 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:50.546080 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:50.547046 (+ 966us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57240: BlockingRecv error: recv got EOF from 127.0.0.1:57240 (error 108)
Metrics: {"server-negotiator.queue_time_us":124}
W20260806 03:05:50.618430 23417 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:05:50.616208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57244 (local address 127.31.74.253:38467)
0806 03:05:50.616452 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:50.616458 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:50.616825 (+ 367us) server_negotiation.cc:407] Connection header received
0806 03:05:50.616998 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:50.617001 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:50.617082 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:50.617186 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:50.618178 (+ 992us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57244: BlockingRecv error: recv got EOF from 127.0.0.1:57244 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
W20260806 03:05:50.752878 23417 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:05:50.750815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57248 (local address 127.31.74.253:38467)
0806 03:05:50.751067 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:50.751074 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:50.751405 (+ 331us) server_negotiation.cc:407] Connection header received
0806 03:05:50.751661 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:50.751667 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:50.751743 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:50.751861 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:50.752662 (+ 801us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57248: BlockingRecv error: recv got EOF from 127.0.0.1:57248 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260806 03:05:51.016114 23417 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 03:05:51.013896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57252 (local address 127.31.74.253:38467)
0806 03:05:51.014130 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:51.014139 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:51.014559 (+ 420us) server_negotiation.cc:407] Connection header received
0806 03:05:51.014828 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:51.014832 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:51.014910 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:51.015023 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 03:05:51.015847 (+ 824us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57252: BlockingRecv error: recv got EOF from 127.0.0.1:57252 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260806 03:05:51.032641 23452 catalog_manager.cc:1259] Loaded cluster ID: d33a3bdbfbcd43f1b19cc835e64dafb7
I20260806 03:05:51.032763 23452 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f35d312a79ef46a2b918f8933914bbe0: loading cluster ID for follower catalog manager: success
I20260806 03:05:51.036060 23452 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f35d312a79ef46a2b918f8933914bbe0: acquiring CA information for follower catalog manager: success
I20260806 03:05:51.038465 23452 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f35d312a79ef46a2b918f8933914bbe0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 03:05:51.075505 23462 catalog_manager.cc:1259] Loaded cluster ID: d33a3bdbfbcd43f1b19cc835e64dafb7
I20260806 03:05:51.075613 23462 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 04659e01c08e47b1a75c59eae0fb432b: loading cluster ID for follower catalog manager: success
I20260806 03:05:51.078217 23462 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 04659e01c08e47b1a75c59eae0fb432b: acquiring CA information for follower catalog manager: success
I20260806 03:05:51.080129 23462 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 04659e01c08e47b1a75c59eae0fb432b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 03:05:51.540433 23549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:51.532636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57254 (local address 127.31.74.253:38467)
0806 03:05:51.533225 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:51.533231 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:51.533492 (+ 261us) server_negotiation.cc:407] Connection header received
0806 03:05:51.533679 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:51.533684 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:51.533860 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:51.533973 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 03:05:51.534559 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:51.535510 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:51.537432 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:51.537937 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:51.538062 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:05:51.538691 (+ 629us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 03:05:51.539402 (+ 711us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 03:05:51.539537 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0806 03:05:51.539709 (+ 172us) server_negotiation.cc:299] Negotiation successful
0806 03:05:51.539803 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":274,"threads_started":1}
I20260806 03:05:51.553928 23555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:51.547456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47524 (local address 127.31.74.252:41723)
0806 03:05:51.548014 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:51.548021 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:51.548242 (+ 221us) server_negotiation.cc:407] Connection header received
0806 03:05:51.548532 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:51.548537 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:51.548702 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:51.548885 (+ 183us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 03:05:51.549371 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:51.550096 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:51.551611 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:51.552026 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:51.552164 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:05:51.552716 (+ 552us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 03:05:51.553296 (+ 580us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 03:05:51.553458 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0806 03:05:51.553521 (+ 63us) server_negotiation.cc:299] Negotiation successful
0806 03:05:51.553608 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":314,"threads_started":1}
I20260806 03:05:51.583640 23549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:51.561754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57262 (local address 127.31.74.253:38467)
0806 03:05:51.561988 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:51.561999 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:51.575427 (+ 13428us) server_negotiation.cc:407] Connection header received
0806 03:05:51.575634 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:51.575643 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:51.575764 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:51.575874 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 03:05:51.577450 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:51.578702 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:51.581239 (+ 2537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:51.581870 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:51.582003 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:05:51.582568 (+ 565us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 03:05:51.582775 (+ 207us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 03:05:51.582886 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 03:05:51.583093 (+ 207us) server_negotiation.cc:299] Negotiation successful
0806 03:05:51.583187 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 03:05:51.583776 23555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:51.561466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47538 (local address 127.31.74.252:41723)
0806 03:05:51.567597 (+ 6131us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:51.567607 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:51.575949 (+ 8342us) server_negotiation.cc:407] Connection header received
0806 03:05:51.576188 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:51.576193 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:51.576277 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:51.576429 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 03:05:51.577077 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:51.578337 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:51.581040 (+ 2703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:51.581730 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:51.581886 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:05:51.582403 (+ 517us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 03:05:51.582633 (+ 230us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 03:05:51.582738 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0806 03:05:51.583250 (+ 512us) server_negotiation.cc:299] Negotiation successful
0806 03:05:51.583336 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5879}
I20260806 03:05:51.584125 23562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 03:05:51.561219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54758 (local address 127.31.74.254:37433)
0806 03:05:51.575977 (+ 14758us) server_negotiation.cc:206] Beginning negotiation
0806 03:05:51.575985 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 03:05:51.576006 (+ 21us) server_negotiation.cc:407] Connection header received
0806 03:05:51.576132 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 03:05:51.576143 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 03:05:51.576310 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 03:05:51.576522 (+ 212us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 03:05:51.577458 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:51.578564 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:51.580704 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 03:05:51.581346 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 03:05:51.581497 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 03:05:51.582122 (+ 625us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 03:05:51.582722 (+ 600us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 03:05:51.582849 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0806 03:05:51.583002 (+ 153us) server_negotiation.cc:299] Negotiation successful
0806 03:05:51.583089 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14622,"thread_start_us":14745,"threads_started":1}
I20260806 03:05:51.591320 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 23187
I20260806 03:05:51.592523 23251 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 03:05:51.718287 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 23187
W20260806 03:05:51.745219 23359 connection.cc:535] server connection from 127.0.0.1:47508 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260806 03:05:51.745606 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 23263
I20260806 03:05:51.747191 23327 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 03:05:51.874346 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 23263
I20260806 03:05:51.900465 32043 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu pid 23341
I20260806 03:05:51.901681 23404 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 03:05:52.029034 32043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ8w1x/build/asan/bin/kudu with pid 23341
2026-08-06T03:05:52Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (6018 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (12106 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (446710 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST